Houston T640/Ciena installed

The Houston T640 was installed in Houston yesterday and the backbone circuits were lit through the site. Our engineers still have some other work to do today to get the Ciena and PCs installed. Overall, things appear to have gone fairly smoothly.

Denver Abilene router shut down

Denver was one of my favorite routers. Mainly because I love Boulder so much and always got to spend some time out that way. I feel a bit like the delivery doctor and the executioner on this one since I helped install it just under four years ago with Caroline. Debbie Montano stole a few moments away from the Qwest corporate offices to see how things were going. Unfortunately, we gave her a camera and posed for a few photos.

The sad thing is that I have too many pictures of me standing next to network equipment with that same cheesy grin on my face, like I just caught a big bass and am holding it up for the camera. At least Caroline looks like she's putting away some tools or something.

Anyway, Denver hasn't been doing much for anyone lately. All networks were moved off by last week and the backbone links weren't in the path for much of anything. It's been turned down, so our engineers can (gingerly) move it to Houston. We need the Denver power controllers in Houston, so we're going to allow Level3 next week to get them installed for us.

Ah, Past Chris. If Future Chris could tell you anything, I'd probably make sure you know how to lace better. And that you should never look away from your hands to pose for a photograph when you're that close to live DC power. Sigh

Los Angeles T640 installed

Tom and Hans were able to get the T640 installed in Los Angeles fairly quickly. The link to Kansas City is up to provide IP connectivity while we wait for the high speed IP drop from Level3. The OC-192 to Seattle didn't come up, but that's likely because we have a fiber backwards in the path. We didn't have light to the Westin building when they were running the fibers in the fiber meet me room, so there was no way to judge the polarity of the cable.

I'm taking the next few days off, but Andrew is going to work on getting that circuit up while I'm out.

Denver is coming out next week. Turning it off on Wednesday night....

Los Angeles! Albuquerque! Denver! The updates! The updates!

I've been getting many a query from you Internet2 upgrade blog junkies asking whether I was dead or had just turned my back on this whole ballyhoo. Fortunately, neither. I'm very much alive and things have very much been moving forward. I had anticipated a bit of a lull over the last few weeks, but I was (sadly) mistaken. Just a rundown of the big things:

Los Angeles
Savvy I2 blog readers will recall that the Los Angeles router was slated to be installed two weeks ago. We pushed that out a week to accomodate for some power controller installations. We really hate doing DC power from the house since we don't control both ends. So, we went out last week. But there was one, glaring problem: no router. There were a whole host of scheduling mishaps between Fedex and Qwest in Kansas City that meant our router didn't get shipped the prior week. We finally got Fedex on-site on Wednesday to ship it overnight to Los Angeles, but fate decided to stick her long ugly nose into things. There were actually two crates at the Kansas City POP. One contained the router that was going to Los Angeles using Fedex Express Freight. (Note the italicizing of Express Freight. Very important.) The other contained the reclaimed flotsam from the Abilene rack that was going Fedex Freight back to Bloomington. Unfortunately, both got picked up by regular Freight, which meant no guaranteed delivery times in Los Angeles. Up until the very end, it looked like things were going to work out, but it wasn't meant to be. Fate 1, Upgrade Team 0. So, we're sending two engineers back out tomorrow to finish the install and commission the node. They're going to miss all the area fireworks, so they deserve a round of applause

Albuquerque
Though the Ciena was installed in Albuquerque in early June, the phone line proved to be more troublesome than we thought. The Qwest phone group is totally separate than the Qwest transport group, meaning that it was essentially two different companies talking to each other. That slowed things down to the typical speed we'd see with most phone line installs. Add the fact that the Albuquerque POP is essentially a bundle of disjoint buildings in the desert, and you have a recipe for slowness. Fortunately, Tom Johnson stayed on top of things and we got the phone line delivered last Tuesday. The cross-connect into our equipment was completed on Friday. In the interim, ESNET graciously helped us out with a Layer2 VPN to the Washington DC POP so we could commission the node. (Recall that we don't turn the Ciena nodes up. Ciena does.) That paved the way to get the University of New Mexico connected and working up to Salt Lake City.

Denver
Aside from UNM, there were a few others on the Denver router, which we plan to pull out next week. Both Front Range Gigapop and the Utah Education Network are up on the new router and passing traffic. There was a bit of scare when the fiber cross-connect in SLC wouldn't allow us to ping the remote ends. Turns out it was a hardware problem with our router card. Got it replaced and everything looks spiffy

***

So what's next for our intrepid crew? Not too much left, really- in terms of physical work. We're yanking Denver next week. It's being sent to Houston where it will be installed two weeks later (to allow time for the power controllers to be installed). After that, there's a pending Ciena install in Seattle, but we don't have a time frame nailed down for that. There's obviously a lot of work left to transition customers to the new network in LA and Houston, but most of that is something that can be done remotely. Of course, we have to pull out the Sunnyvale, LA and Houston routers at some point in the next few months. I'm working on solidifying that schedule right now.