Now I'll stop posting on Atlanta
And now, drumroll, please....
Atlanta.
Is.
Migrated.
Yeah, I can hardly believe it. The past few days have been up and down (mostly down). Yesterday, I'm ashamed to say, I forgot that the GigE port on the M5 that we were swinging the SOX dark fiber to was LC. Of course, the SOX card has SC connectors and the Qwest techs couldn't locate any SC-SC barrel connectors. So, we had to move back the migration to this evening, while we overnighted an SC-LC jumper and connectors to Atlanta.
Trouble is, we got our wires cross and accidentally sent it to the Level3 POP in Atlanta. We were able to locate a local courier that could drive it between the two POPS, so that turned out to be a non-issue, but one that had me on pins and needles for a few hours.
The next problem was port the fiber was plugged into in the Level3 POP. When Jeff was in Atlanta two weeks ago, I had forgotten that we slotted the LH SFP into port 1 of the 4-port GigE card instead of port 0, like I had planned and wrote in the interface description. When I devised the instructions to Level3 for running the fiber on that side on Monday, I put the wrong port down and completely missed it. Fortunately, while away from the office today at 3, the thought magically popped into my head and I had a coworker double check my work. Turns out, I was indeed mistaken, and we had a short window to get Level3 out to the POP. It was 4:30 and we were still waiting on both techs to call in. Fortunately they did...at the same time.
Magically, the connection came up. I say magically, because at this point, I'd probably believe you if you were to tell me there was imps running around in our fibers just trying to keep Atlanta as-is.
All that was left was to migrate USF down to the M5 at 10PM and do the backbone move at 11PM. The USF move went well, though there was some sort of problem with the adjacent PIC to the University of Mississippi. For some reason, all the BGP sessions through that ATM card went down and IP reachability was nil. None of this work should have affected them, but I reset the card anyway. That fixed it for some reason. Hmmm. Need to check that out in the AM.
The backbone move went as smooth as butter. Down on one router, back up on the other within 2 minutes. Even the Qwest techs were amazed at how easy it was.
So, this feels a bit like an Oscar speech, but we really have to extend our thanks to a lot of people. Scott Friedrich at SOX, and all their work really made the difference. Qwest did an awesome job of testing the circuit in advance and working around the multiple changes in our schedule. Level3 was spot-on with the on-demand remote hands and help today with the courier. This was a real team effort and one of those war stories that will linger in my mind forever.
Onward to Kansas City!
Atlanta.
Is.
Migrated.
Yeah, I can hardly believe it. The past few days have been up and down (mostly down). Yesterday, I'm ashamed to say, I forgot that the GigE port on the M5 that we were swinging the SOX dark fiber to was LC. Of course, the SOX card has SC connectors and the Qwest techs couldn't locate any SC-SC barrel connectors. So, we had to move back the migration to this evening, while we overnighted an SC-LC jumper and connectors to Atlanta.
Trouble is, we got our wires cross and accidentally sent it to the Level3 POP in Atlanta. We were able to locate a local courier that could drive it between the two POPS, so that turned out to be a non-issue, but one that had me on pins and needles for a few hours.
The next problem was port the fiber was plugged into in the Level3 POP. When Jeff was in Atlanta two weeks ago, I had forgotten that we slotted the LH SFP into port 1 of the 4-port GigE card instead of port 0, like I had planned and wrote in the interface description. When I devised the instructions to Level3 for running the fiber on that side on Monday, I put the wrong port down and completely missed it. Fortunately, while away from the office today at 3, the thought magically popped into my head and I had a coworker double check my work. Turns out, I was indeed mistaken, and we had a short window to get Level3 out to the POP. It was 4:30 and we were still waiting on both techs to call in. Fortunately they did...at the same time.
Magically, the connection came up. I say magically, because at this point, I'd probably believe you if you were to tell me there was imps running around in our fibers just trying to keep Atlanta as-is.
All that was left was to migrate USF down to the M5 at 10PM and do the backbone move at 11PM. The USF move went well, though there was some sort of problem with the adjacent PIC to the University of Mississippi. For some reason, all the BGP sessions through that ATM card went down and IP reachability was nil. None of this work should have affected them, but I reset the card anyway. That fixed it for some reason. Hmmm. Need to check that out in the AM.
The backbone move went as smooth as butter. Down on one router, back up on the other within 2 minutes. Even the Qwest techs were amazed at how easy it was.
So, this feels a bit like an Oscar speech, but we really have to extend our thanks to a lot of people. Scott Friedrich at SOX, and all their work really made the difference. Qwest did an awesome job of testing the circuit in advance and working around the multiple changes in our schedule. Level3 was spot-on with the on-demand remote hands and help today with the courier. This was a real team effort and one of those war stories that will linger in my mind forever.
Onward to Kansas City!
posted by Chris Robb at 5/04/2007 02:50:00 AM
<< Home