The long and short on Atlanta
Well, as most of you have surmised from the I2 notifications that went out, the work in Atlanta did not go as planned last night. The short answer is that it came down to a bad optical splitter, thus halting us at step 1 of the 8-step transition I had planned out. Because of this, Washington DC needs to stick around for at least another week. For those that are wanting to hear the saga, read on...
We got confirmation that all the cross-connects had been run in the 56 Marietta FMMR on Tuesday. I had Level3 slot cards for me on Wednesday to do some pre-testing. We immediately ran into problems. There was some confusion on the panel assignments in the FMMR that were resolved early Thursday morning. So, we spent all day Thursday working with SOX and Qwest on getting the fibers cleaned up to the point where we could continue with the transition later in the evening.
First, on to Qwest. They've been absolutely fantastic in keeping in touch with us on getting things pre-tested. Once we were ready on our end, we plugged the OC-192 into our router and were testing to a loop. A bunch of us were under the impression that there was active electronics at 55Marietta. When the network was initially installed in 2003, there was a Ciena box at 55M that was extended over the long metro jump to Forest Park. Somewhere along the line that changed, and the circuit no longer passed through 55M. We didn't discover this until I got on the phone with the tech and he said the path was glass all the way through 55M to Forest Park, some 12 miles away. It was no wonder I couldn't see the hard loop they placed for me. I couldn't shoot that distance there and back. The put a test set on it and we did a head-to-head test. I could see them and they could see me, though we're right on the receive sensitivity margin. The circuit was 100% clean, though, so all looked well. First crisis averted.
The SOX connection was a day-long struggle to remove attenuation from the patches between Level3 and Georgia Tech. To test things out, we temporarily plugged a 10GigE LR interface directly into the metro fiber on both sides, bypassing the optical splitter. The initial reading on the Georgia Tech side was -13dB- enough to get the link up, but probably not enough to shoot through the attenuation that would come with the insertion of the splitters. By days end, we had the signal strength up to -9dB. Second crisis averted.
The Internet2 engineer that was going to be stationed at Level3 during the change window phoned me up at 4PM and said he couldn't find the optical splitter in our suite. While he was locating a tech, I was mapping out directions to drive down to Atlanta to get it to him early in the morning. Fortunately, they located it just before 5PM and all was well. Third crisis averted.
So, I went home and spent the next three hours pre-configuring the routers with all the interface, BGP, MSDP, etc configuration that needed to be done. 10PM rolls around and everyone dials into the bridge. I don't know how many of you have ever had a bunch of people in colo facilities on a phone bridge at once, but it's almost impossible to talk. Jeff and I had to connect on a separate phone line since we couldn't hear each other.
We started out with the first step: moving SOX over to the optical splitter. No link. -36dB received on the I2 side. We did a bunch of different testing, including removing the splitter from the path, swapping out all the fibers, and jiggling connectors. In the end, the splitter was adding 9-10dB of attenuation in both directions. Well, it isn't supposed to be doing that, but of course, we had no other way of proceeding. So, we backed out of everything. Cancelled the SOX, M5 and USF move with Qwest. Cancelled the backbone circuit move. Cancelled the Washington DC router pullout (and called the engineers who were going to get up early in the Friday AM to fly out).
We have some alternate plans. There's actually a third pair of fiber we ordered a while ago that should be ready for our use if we can get the tie-down info in time to order a cross-connect. We can also shim HP switches in the middle and plumb the GigE through SOX. (SOX, unfortunately, doesn't have any long-haul 1G optics for their Force10, which was something we considered a while ago) I'm hopeful that we can get the fiber since that's the easiest way with the least moving parts, but if we need to use something else, we will.
Right now, I'm aiming for a transition next Wednesday night. Once that gets more solid, I'll let everyone know.
We got confirmation that all the cross-connects had been run in the 56 Marietta FMMR on Tuesday. I had Level3 slot cards for me on Wednesday to do some pre-testing. We immediately ran into problems. There was some confusion on the panel assignments in the FMMR that were resolved early Thursday morning. So, we spent all day Thursday working with SOX and Qwest on getting the fibers cleaned up to the point where we could continue with the transition later in the evening.
First, on to Qwest. They've been absolutely fantastic in keeping in touch with us on getting things pre-tested. Once we were ready on our end, we plugged the OC-192 into our router and were testing to a loop. A bunch of us were under the impression that there was active electronics at 55Marietta. When the network was initially installed in 2003, there was a Ciena box at 55M that was extended over the long metro jump to Forest Park. Somewhere along the line that changed, and the circuit no longer passed through 55M. We didn't discover this until I got on the phone with the tech and he said the path was glass all the way through 55M to Forest Park, some 12 miles away. It was no wonder I couldn't see the hard loop they placed for me. I couldn't shoot that distance there and back. The put a test set on it and we did a head-to-head test. I could see them and they could see me, though we're right on the receive sensitivity margin. The circuit was 100% clean, though, so all looked well. First crisis averted.
The SOX connection was a day-long struggle to remove attenuation from the patches between Level3 and Georgia Tech. To test things out, we temporarily plugged a 10GigE LR interface directly into the metro fiber on both sides, bypassing the optical splitter. The initial reading on the Georgia Tech side was -13dB- enough to get the link up, but probably not enough to shoot through the attenuation that would come with the insertion of the splitters. By days end, we had the signal strength up to -9dB. Second crisis averted.
The Internet2 engineer that was going to be stationed at Level3 during the change window phoned me up at 4PM and said he couldn't find the optical splitter in our suite. While he was locating a tech, I was mapping out directions to drive down to Atlanta to get it to him early in the morning. Fortunately, they located it just before 5PM and all was well. Third crisis averted.
So, I went home and spent the next three hours pre-configuring the routers with all the interface, BGP, MSDP, etc configuration that needed to be done. 10PM rolls around and everyone dials into the bridge. I don't know how many of you have ever had a bunch of people in colo facilities on a phone bridge at once, but it's almost impossible to talk. Jeff and I had to connect on a separate phone line since we couldn't hear each other.
We started out with the first step: moving SOX over to the optical splitter. No link. -36dB received on the I2 side. We did a bunch of different testing, including removing the splitter from the path, swapping out all the fibers, and jiggling connectors. In the end, the splitter was adding 9-10dB of attenuation in both directions. Well, it isn't supposed to be doing that, but of course, we had no other way of proceeding. So, we backed out of everything. Cancelled the SOX, M5 and USF move with Qwest. Cancelled the backbone circuit move. Cancelled the Washington DC router pullout (and called the engineers who were going to get up early in the Friday AM to fly out).
We have some alternate plans. There's actually a third pair of fiber we ordered a while ago that should be ready for our use if we can get the tie-down info in time to order a cross-connect. We can also shim HP switches in the middle and plumb the GigE through SOX. (SOX, unfortunately, doesn't have any long-haul 1G optics for their Force10, which was something we considered a while ago) I'm hopeful that we can get the fiber since that's the easiest way with the least moving parts, but if we need to use something else, we will.
Right now, I'm aiming for a transition next Wednesday night. Once that gets more solid, I'll let everyone know.
posted by Chris Robb at 4/27/2007 03:37:00 PM
<< Home