Did you miss it? Salt Lake City installed last week
Apparently our network elves have been hard at work in the wee hours of the morning cobbling shoes and installing router nodes. Unbeknownst to many, the Internet2 network peeked its head into the Beehive State and decided to set up shop. It's safely tethered to the Kansas City router now, but we're aiming to change that in the next week with a second lifeline to Seattle. That will make it a full-fledged grown-up router, able to route with the best of them.
Steve Corbato dropped by on Friday just to make sure our boys hadn't electrocuted themselves and posted some photos of the visit. I know what you're going to ask, and the answer is: "No, the POP doesn't have a brew-pub in one of the suites."
For some reason the router still thinks its in Atlanta. :-) I''ll be configuring it tomorrow so it can play in all the reindeer games with the rest of the big boys.
We have two teams heading out West and South to install Denver and Baton Rouge Ciena nodes. We wish them well.
Kansas City backbone and GPN transitioned
I'm happy to announce that the Kansas City-Chicago and Kansas City-Atlanta OC-192s are fully in production on the new network and are routing traffic between the three cities. In addition, we now have a new interconnect between the old Abilene network and the new Internet2 network in Kansas City.
Concurrent with that, GPN migrated their 10GigE over to the Level3 POP.
We expect to have the Salt Lake City T640 up next week with connectivity back to Kansas City and (possibly) to Seattle.
Have a good weekend.
KANS T640, GPN transition, El Paso, Atlanta
Rest assured, we've been diligently working this week: on not-so-public/exciting/sexy install packets, shipping and documentation. It's all been prep for today, when four major tasks hit.
The Kansas City T640 is fully configured and has backbone links to Atlanta and Chicago. It's been brought into the iBGP mesh and I'm putting the finishing touches on the monitoring/documentation. The Chicago circuit came up yesterday, and the Houston circuit (actually two circuits glued together in Houston in anticipation of the eventual Houston T640 install) was turned up 10 minutes ago.
Second: we nailed down the logistics of the GPN transition to the new network tomorrow. GPN will be moving their router from Qwest to Level3, thus abandoning some dark fiber between the two POPs. Being the opportunistic people that we are, we're both using that fiber to maintain some connectivity into the Qwest POP using the same CWDM splitters we used in Atlanta. The clock starts ticking at 9AM tomorrow on the move. Our side is fairly straightforward, and I'm sending good thoughts toward the local engineers that have to lug the 7600 between POPs.
Third: The El Paso Ciena install is underway. All equipment is on site and I heard good things from the field on the progress made. That should wrap up tonight or early tomorrow morning.
Fourth: The Atlanta T640 was pulled out of the Qwest space and will shortly begin its wonderful journey across the country to Salt Lake City for install next week.
I love it when a plan comes together.
Kansas City Install almost complete
With the exception of a few power runs and a bit of documentation, the Kansas City install is largely complete at the Level3 POP. The T640 arrived yesterday and was powered up. We'll be activating the backbone circuit to Chicago today. The circuit to Atlanta (temporarily bypassing Houston until it gets installed) will have to wait for Monday when we slot the OC-192 I had yanked from New York on Thursday).
We're solidifying the details, but there will be an interconnect between the old Abilene router and the new Internet2 Network router via some University of Nebraska metro fiber. We'll be putting another one of those CWDM optical combiner/separators on the link to share the path with GPN. Not sure on the timing of that yet, but it's starting to look like we'll make that happen next Friday. Once that's complete, we'll drop the Qwest OC-192 between the Abilene Kansas City router and the Internet2 router in Chicago. That frees up some much-needed OC-192 PICs and allows us to move forward in Salt Lake City in two weeks' time.
DC router removal saga
You may be wondering why I didn't gush over the perfect router removal we had on Monday. That's because it wasn't meant to be. Our engineers' flight to DC got cancelled on Monday morning and they couldn't get rebooked until later in the afternoon. So, they rented a car, put on some tunes and did a Tommy-Boy road trip out to DC. Knowing Jay, there was probably a lot of Rush involved.
Jay and AJ got to the Qwest POP this morning only to find that the empty T640 crate had been "obliterated" in shipping. This is where the driving came in handy. They had a smaller T320 crate that they were able to use for the T640, and the put the rest of the de'commed equipment in their Taurus. Sadly, no more room for Rush.
Unfortunately, the work at Qwest took them pass their badging window at Level3, so they need to stick around another day so they can complete some PC install work early tomorrow morning. Then it's back home to Indiana.
We're attempting to secure another T640 shipping crate from Juniper, but if anyone has one that they've been using as a doghouse or Tornado shelter, drop me a line. I'm off to check Craigslist.
Atlanta and Washington DC Abilene T640s turned down
I turned down both the Atlanta and Washington DC T640s just before 1PM EDT this afternoon. This officially removes them from the network, though they stopped routing traffic last week when we moved the Atlanta-Houston circuit over to the new router in Atlanta.
Washington DC is being pulled out tomorrow. It was originally slated for today, but our engineers' flight to DC was cancelled and they won't be able to get in before 5PM this afternoon. That router will make a quick trip to Kansas City for install on Wednesday.
Jacksonville Ciena installed
Just a quick note that I got a call from Tom Johnson to tell me that they had finished the Ciena install in Jacksonville.
Now I'll stop posting on Atlanta
And now, drumroll, please....
Atlanta.
Is.
Migrated.
Yeah, I can hardly believe it. The past few days have been up and down (mostly down). Yesterday, I'm ashamed to say, I forgot that the GigE port on the M5 that we were swinging the SOX dark fiber to was LC. Of course, the SOX card has SC connectors and the Qwest techs couldn't locate any SC-SC barrel connectors. So, we had to move back the migration to this evening, while we overnighted an SC-LC jumper and connectors to Atlanta.
Trouble is, we got our wires cross and accidentally sent it to the Level3 POP in Atlanta. We were able to locate a local courier that could drive it between the two POPS, so that turned out to be a non-issue, but one that had me on pins and needles for a few hours.
The next problem was port the fiber was plugged into in the Level3 POP. When Jeff was in Atlanta two weeks ago, I had forgotten that we slotted the LH SFP into port 1 of the 4-port GigE card instead of port 0, like I had planned and wrote in the interface description. When I devised the instructions to Level3 for running the fiber on that side on Monday, I put the wrong port down and completely missed it. Fortunately, while away from the office today at 3, the thought magically popped into my head and I had a coworker double check my work. Turns out, I was indeed mistaken, and we had a short window to get Level3 out to the POP. It was 4:30 and we were still waiting on both techs to call in. Fortunately they did...at the same time.
Magically, the connection came up. I say magically, because at this point, I'd probably believe you if you were to tell me there was imps running around in our fibers just trying to keep Atlanta as-is.
All that was left was to migrate USF down to the M5 at 10PM and do the backbone move at 11PM. The USF move went well, though there was some sort of problem with the adjacent PIC to the University of Mississippi. For some reason, all the BGP sessions through that ATM card went down and IP reachability was nil. None of this work should have affected them, but I reset the card anyway. That fixed it for some reason. Hmmm. Need to check that out in the AM.
The backbone move went as smooth as butter. Down on one router, back up on the other within 2 minutes. Even the Qwest techs were amazed at how easy it was.
So, this feels a bit like an Oscar speech, but we really have to extend our thanks to a lot of people. Scott Friedrich at SOX, and all their work really made the difference. Qwest did an awesome job of testing the circuit in advance and working around the multiple changes in our schedule. Level3 was spot-on with the on-demand remote hands and help today with the courier. This was a real team effort and one of those war stories that will linger in my mind forever.
Onward to Kansas City!
Last night's fiber cut
Now that the dust has settled on last night's bridge fire, we'd like to extend our gratitude to Level3 for making such a rapid repair. We've dealt with bridge/tunnel/road/railway fires before, and this was probably one of the quickest repair times we've seen for such a disastrous event. This also highlights the flexibility we have to route traffic around such events. Kudos to all involved.
Boston circuits update
Level3 rolled everyone to a new fiber path in Boston shortly after the last update. All circuits on that path are back up and working. I believe the response time was just under 4 hours. We know they migrated more than just the Internet2 Infinera network over to alternate fiber paths. Pretty impressive footwork on pretty short order to get around a severe problem.
Thank You for Not Smoking.
Update on the Boston bridge fire
While working through the motions of getting the Boston waves routed around the other side of the ring via DC, Level3 informed us that the outlook on the downtime isn't as bad as they initially expected. They're working toward routing their network across some diverse fiber, as well as trying to fix the fiber under the bridge. The prognosis is now on the order of hours, and not days. We're expecting to have partial or complete restoration in the next several hours.
SOX migrated in Atlanta
All of the networks behind SOX have been successfully migrated in Atlanta. Tomorrow we'll be swinging the Houston OC-192 from the Atlanta Qwest router to the Atlanta Level3 router.
Question: When can a cigarette take down your network?
Answer: When you throw it at a bridge and light it on fire.
Looks like all our circuits between Boston and New York are down due to a fire under the Longfellow Bridge in Boston. Read all the fun here: http://wbztv.com/topstories/local_story_121202334.html
Artists rendering of the moment our circuits went down
There were a few updates after this:
Last Night's Fiber Cut
Looking better in Atlanta
We're taking another stab at Atlanta this week. We were hoping for a third pair of metro fiber to be delivered in time for the new attempt. When that didn't look promising, we tested out a new pair of optical splitters last Friday and overnighted them to Level3 and SOX. I just got off the phone with both Scott Friedrich (Georgia Tech) and Level3. The new splitters have been inserted and we brought up link to a second 10GigE LR port on the SOX Force10. Here's the next order of events:
- Tonight, I'll be migrating SOX over to the new router. They've already pre-configured the VLANs on the new port, so it's just a matter of deactivating one config and turning down another.
- Tomorrow, during the day, I'll work with Qwest to move the abandoned SOX fiber over to the M5 to bring up the GigE LH link to the new router.
- Tomorrow, during the night, I'll be switching the M5 over to the new fiber, migrating the USF OC-3 down to the M5 and moving the Qwest HSTN-ATLA backbone circuit to the new router.
I'm starting to get superstitious about these sorts of things, so if I could ask everyone in the R/E community to cross their fingers for us, I think we'll be OK. If things don't work, I'm going to blame YOU! :-)
