The long and short on Atlanta

Well, as most of you have surmised from the I2 notifications that went out, the work in Atlanta did not go as planned last night. The short answer is that it came down to a bad optical splitter, thus halting us at step 1 of the 8-step transition I had planned out. Because of this, Washington DC needs to stick around for at least another week. For those that are wanting to hear the saga, read on...

We got confirmation that all the cross-connects had been run in the 56 Marietta FMMR on Tuesday. I had Level3 slot cards for me on Wednesday to do some pre-testing. We immediately ran into problems. There was some confusion on the panel assignments in the FMMR that were resolved early Thursday morning. So, we spent all day Thursday working with SOX and Qwest on getting the fibers cleaned up to the point where we could continue with the transition later in the evening.

First, on to Qwest. They've been absolutely fantastic in keeping in touch with us on getting things pre-tested. Once we were ready on our end, we plugged the OC-192 into our router and were testing to a loop. A bunch of us were under the impression that there was active electronics at 55Marietta. When the network was initially installed in 2003, there was a Ciena box at 55M that was extended over the long metro jump to Forest Park. Somewhere along the line that changed, and the circuit no longer passed through 55M. We didn't discover this until I got on the phone with the tech and he said the path was glass all the way through 55M to Forest Park, some 12 miles away. It was no wonder I couldn't see the hard loop they placed for me. I couldn't shoot that distance there and back. The put a test set on it and we did a head-to-head test. I could see them and they could see me, though we're right on the receive sensitivity margin. The circuit was 100% clean, though, so all looked well. First crisis averted.

The SOX connection was a day-long struggle to remove attenuation from the patches between Level3 and Georgia Tech. To test things out, we temporarily plugged a 10GigE LR interface directly into the metro fiber on both sides, bypassing the optical splitter. The initial reading on the Georgia Tech side was -13dB- enough to get the link up, but probably not enough to shoot through the attenuation that would come with the insertion of the splitters. By days end, we had the signal strength up to -9dB. Second crisis averted.

The Internet2 engineer that was going to be stationed at Level3 during the change window phoned me up at 4PM and said he couldn't find the optical splitter in our suite. While he was locating a tech, I was mapping out directions to drive down to Atlanta to get it to him early in the morning. Fortunately, they located it just before 5PM and all was well. Third crisis averted.

So, I went home and spent the next three hours pre-configuring the routers with all the interface, BGP, MSDP, etc configuration that needed to be done. 10PM rolls around and everyone dials into the bridge. I don't know how many of you have ever had a bunch of people in colo facilities on a phone bridge at once, but it's almost impossible to talk. Jeff and I had to connect on a separate phone line since we couldn't hear each other.

We started out with the first step: moving SOX over to the optical splitter. No link. -36dB received on the I2 side. We did a bunch of different testing, including removing the splitter from the path, swapping out all the fibers, and jiggling connectors. In the end, the splitter was adding 9-10dB of attenuation in both directions. Well, it isn't supposed to be doing that, but of course, we had no other way of proceeding. So, we backed out of everything. Cancelled the SOX, M5 and USF move with Qwest. Cancelled the backbone circuit move. Cancelled the Washington DC router pullout (and called the engineers who were going to get up early in the Friday AM to fly out).

We have some alternate plans. There's actually a third pair of fiber we ordered a while ago that should be ready for our use if we can get the tie-down info in time to order a cross-connect. We can also shim HP switches in the middle and plumb the GigE through SOX. (SOX, unfortunately, doesn't have any long-haul 1G optics for their Force10, which was something we considered a while ago) I'm hopeful that we can get the fiber since that's the easiest way with the least moving parts, but if we need to use something else, we will.

Right now, I'm aiming for a transition next Wednesday night. Once that gets more solid, I'll let everyone know.

Washington DC almost definitely coming out on Friday

Got another bit of good news. The FMMR cross-connect between Qwest and Georgia Tech has been completed, which means all the patching is in place for the Atlanta-Houston OC-192 groom on Thursday night. We've scheduled travel and I'm arranging to do some testing. Still need to verify the end to end path, but we're headed in the right direction.

University of Louisville 10GigE up on Atlanta

Yeah, so I'm stuck in the Indianapolis airport waiting on my delayed flight to Washington for the I2MM when I got a note that made me very happy. Caren tells me that the University of Louisville's 10GigE connection from Louisville to Atlanta is up and passing traffic. This was one of those connections that ate up a lot of space on our e-mail server, so it's one that I'm happy to see completed.

Now, if my plane would just arrive the day would be complete.

Tulsa Ciena and DC removal

The installs have resumed a bit this week with Tom and Hans slotting the chassis today. So far, so good, as I've not heard of any issues from the field. With luck, they'll be finished by day's end and ready to come back.

We're currently targeting next Friday (4/27) for the Washington DC removal. We did receive the final LOA/CFA we needed from Qwest yesterday in Atlanta, but there were too many unknowns in terms of scheduling and heroics needed to get crates shipped around that we decided to not to move the Atlanta backbone circuit this week. With the Members Meeting next week, we decided that the earliest we could schedule it would be next Thursday. Hopefully the move will go well and we'll be able to yank the DC chassis the next morning. That will put us on track to install Kansas City the following week.

Washington DC node removal status

I've gotten a few questions from the community about how the Washington DC router removal is going. We're unfortunately still waiting on some documentation and an order to process for a FMMR cross-connect. While we continue to reach out to our partners, I don't have any ETA on when that will be ready. For now, the IP network transition is on hold until we can get that jumper run. Funny how a 2 meter piece of fiber can hold you up....

We're still looking into alternative transport methods and am getting more inventive as we go. Hope to have more shortly. At this point, I'm doubtful it will happen this week. Even if we can get the proper LOA/CFA, it will likely take several days to get the cross-connect made in the FMMR. It also takes a minimum of 72 hours notice to get remote hands lined up- none of which I can easily arrange until I have some level of confidence that the jumpers will be run.

But hey, I've been wrong before. Maybe things will start to click.

So why hasn't the Washington DC router moved yet?

In a word: Atlanta.

Have a look at this image to the right, which depicts the current state of the network, with both the Qwest and Level3 backbone links. Note that the old Abilene network (with routers in blue) and the new Internet2 network (with routers in red) are interconnected in New York and Chicago. Those two links are keeping the two networks talking to each other.

If we were to remove the Washington DC router, we would also have to turn down the DC to New York Qwest OC-192, which would mean the two networks are only interconnected in Chicago. Unacceptable.

A few months back, we looked at two different options to keep the networks dual-homed. The first was to do something similar to what we did in Indianapolis: have Qwest patch the southern route into a single Houston to New York circuit. The second option was to figure out a way to create an interconnect in Atlanta. Because we didn't like the idea of traffic having to go all the way to New York to get between the two networks, we focused on interconnecting in Atlanta. Stay with me.

Atlanta has a few additional quirks that make the transition a bit more difficult. Ever since the 2002 migration to the Juniper platform, we've had a small Juniper M5 providing connectivity for the University of Mississippi OC-3 ATM (the T640 platform doesn't have small ATM interfaces). Mississippi isn't quite ready to move on our schedule yet, so we need to leave the M5 at the Qwest POP, with the Mississippi circuit still connected. So, we need to interconnect that M5 with the new T640 at Level3. There's a good 20 miles of space between the two POPs.

Fortunately, Georgia Tech is abandoning a pair of fiber between 56 Marietta and Forest Park (Qwest). Internet2 has two pair of metro fiber between 56 Marietta and 180 Peachtree (Level3). Georgia Tech's connection will migrate to the Internet2 metro fiber, thus freeing us up to use their old 20 mile run out to Qwest from 56M.

The original plan was to put the M5 GigE interconnect on one pair of fiber and SOX on the other pair. But then we realized that we could swing the Qwest Houston to Atlanta circuit up from the Qwest POP in Atlanta to the Level3 POP in Atlanta. That circuit passes near 56 Marietta, so we started down the path of figuring out how to cram three signals onto two pair of fiber. The solution was a wideband CWDM passive optical splitter. The SOX and M5 GigE will share a pair of fiber and the Qwest circuit will ride the other.

Even though I know that's crystal clear and no one could possibly be confused, I made up a somewhat irrelevant diagram to explain the obvious. :-)

Now that that is crystal clear, I'll explain the holdup. Raise your hand if you've ever had a fiber meet me room (FMMR) cross-connect work perfectly the first time around. I see five hands. When I discount those of you that are lying, I see one- and that person probably works in the FMMR. It's typical to encounter delays on getting panels right and making sure the person has the appropriate authorization to plug a fiber in. Each carrier wants it done a little differently. For example, Qwest wants the tech to dangle the fiber, but not plug it in. You need to open a ticket to have Qwest go out and plug the fiber in. None of this is particularly wrong- just difficult to navigate sometimes. That's what we're running into here.

We were hoping to do all the transition in Atlanta last night, but we've pushed it back to next week. That will delay the Washington DC router removal by a week. Since the DC router is being shipped to Kansas City, the KC install will likely also be pushed back. We have a bit of cushion on the schedule, so it's possible that things may accelerate past that, but probably unlikely.

So there you have it. I was clearly misinformed when I believed that Chicago was going to be the most complex transition in the network. That's looking like a campus wiring closet switch install compared to Atlanta. :-)

NGIX-East migrated to new Washington DC T640

Caren moved all the NGIX-East peerings from the old Washington DC router at the Qwest POP to the new Washington DC router. All went well.

Along with the Drexel and PSC moves earlier in the week, that means everyone is off the old router and it can be turned down...

Well, almost. I'll explain in a bit...

Chicago to New York IP Backbone Link up !

Yesterday afternoon I brought a new OC-192 IP backbone link over the Level3/Infinera system between Chicago and New York via Cleveland and Boston. After our standard testing, this was put into production around 1pm today. The fiber path for this circuit is actually longer than the fiber path from Chicago to Washington D.C. to New York. Since we use the fiber route miles as our IS-IS metrics to provide the lowest latency path, this will only be used as a backup path for now.

This connection is important to the transition because it allows us to remove the old Washington T640 without leaving the New York T640 single-homed.

Boston Ciena node installed

The Boston Ciena node was successfully installed yesterday by engineers Andrew Lee and Hans Addleman. With the exception of a bloody mishap with the tools, everything went well. :-) Ciena technicians are now able to come in and turn the node up.

Indianapolis T640 removed from service

Matt Davy was successful in turning down the two remaining backbone circuits that pass through the Indianapolis T640 this morning. The old KSCY-IPLS and IPLS-CHIN circuits are now fused together in Indianapolis to create a Kansas City to Chicago Qwest circuit. The Indianapolis T640 is now completely disconnected from the network and will ship back to Juniper this Monday. Another round of thanks to Qwest for their assistance this morning.

U of Louisville temporary connection to the Indiana Gigapop

The time has finally come for the loaner T640 router to return back to its home. As many of you know (don't you!) that router's been passed around the network a few times, where its been needed. It's been providing connectivity in Indianapolis for the past few months, while we sent the old Indianapolis router to Washington DC.

Early this morning, Caren and Jay moved the last connector, the University of Louisville, off the Indy router. While their permanent connection will be a 10GigE to Atlanta, they're still waiting on some fiber work to be completed in Louisville. Fortunately, they agreed to temporarily move behind the Indiana Gigapop so as not to hold of the return of the Juniper router. With Monday's migration of the Indiana Gigapop to Atlanta, we freed up a pair of fiber between the Gigapop and Qwest. Louisville was regroomed to that pair. It's everyone's hope that this only lasts a short time. While, we're happy to host them, we know they're anxious to get moved over to their dedicated connection.

We should pause and give our extreme thanks to Juniper for their generous assistance over the past five months. We wouldn't have been able to move the networks in place without their support.

The Indiana Gigapop migrated to Atlanta

This past Monday evening/Tuesday morning, Matt Davy successfully migrated the Indiana Gigapop to its new 10GigE interconnect to the new Atlanta Internet2 router. This frees up a pair of fiber from the Indiana Gigapop to Qwest that will be temporarily used by the University of Louisville.