Tuesday, October 10, 2006


We've gone back and forth about the best method to provide out-of-band access at the new network. The big router sites are mostly taken care of with some high speed IP drops from Level3. (The exceptions are Chicago, because it's off-net, and Seattle and NYC, where our routers aren't going into L3 space) The rest of the network is left to fend for itself in the event of an outage....

Of course, we wouldn't leave a grooming node or remote router site just sitting by it's lonesome during an outage with nothing to do but give us headaches. We looked at a lot of different ways to provide fail-safe access:

  • in-band STS channel with access from both sides of a redundant ring- too risky if the control plane blasts the management channel away
  • high speed IP drops with IPSEC tunnels to IU - not available in some sites; expensive
  • DSL - difficult to order; DSLAM hardware and firmware compatibility issues; some carriers won't let you terminate in your DSL modem
  • ISDN - low bandwidth; difficult to troubleshoot and order
  • POTS line - low bandwidth

Eventually we settled on a hybrid plan whereby we use the in-band STS channel with fiber path redundancy for protection. We'll get POTS lines at all locations as a fall-back. We know that won't give us the bandwidth we need to manage the grooming/DWDM boxes through their GUI interfaces (at least, it would make things abysmally slow), but we can bring the STS management channel back up using canned TL-1 commands that we keep handy in the event of a catastrophic outage. POTS is also easier to troubleshoot than ISDN. Pick up the phone and you either get a dial-tone or you don't. See carrier for any further problems

Now the challenge is going to be getting the last cross-connect between the carrier's punchdown block in the POP to our suite. Level3 wants a DLR from the carrier, which can be challenging to get when talking to your average sales rep. Need to approach that issue quickly. I may hand it off to someone else so it doesn't block on my time.

We initially thought it would be useful to stick a phone in the POP. The Cisco interface card we're looking at using has a second RJ-11 jack on the back for an extension. Caren brought up the excellent point that life would be bad if we had to rely on the POP techs to hang up the phone when they were finished using it during a hands and eyes request. We'll settle with sending our install teams out with a handset they can plug in.