It's been fun....

As some of you already know, the Internet2 transition to Level3 has been complete for the past few weeks. The first clue was probably all the remaining Abilene nodes disappearing from the weathermap last week. They've been shut down and the first is coming out tomorrow. The other nodes will be removed over the next few weeks. 

With everything complete, these little updates will cease. This was a wonderful experiment into the benefits (and perils) of direct communication between the Internet2 NOC engineering staff and the community. We learned a bit and may do more in the future in a more ongoing operational capacity, but don't look for that to happen anytime soon. 

There are really too many people behind the scenes to thank. I'd like to acknowledge several individuals in particular that spent many sleepless nights on the road helping to make this transition a success. These are the true heroes of the upgrade that really made this all come together. 

• Hans Addleman
  
• Andrew Lee
  
• Caroline Carver
  
• Jay Duncan
  
• Tom Johnson
  
• Aaron Pagel
  
• AJ Ragusa

In addition the following people found other ways to assist. My thanks go out to them as well:

• Steve Peck and Nathan Lucas for keeping the NOC service desk in the loop and assisting with procedural work
  
• Matt Davy for pinch hitting for me when I was out in late March and keeping the CPS network on track throughout the upgrade
• Caren Litvanyi for wading through countless spreadsheets and analyzing the minutiae of the suite buildouts
• Luke Fowler and Ed Balas for their support on the database and documentation systems
• Chris Small for keeping the observatory in step
• John Graham for managing the entire Ciena network and alleviating hundreds of hours of concern
• Jon-Paul Herron for keeping everyone's sanity at bay

This truly was a team effort and I'm proud to work with such high caliber individuals. If you see them in the hall at a conference, give them a pat on the back. They all deserve it. 

So long....

When one problem exposes another

Sometimes the network Gods have a funny way of toying with your mind. We noticed some L3 incompletes on the new CalREN circuit yesterday, and we pointed the finger at the optical levels on the CalREN interface. I worked with Level3 early this morning to clean the fiber path between I2 and CalREN. They found a bad port in the path at 600 Wilshire that was causing us to launch -14.3dB toward CalREN- just a hair under the tolerance specs of their card. After we cleaned that up, the L3 incompletes persisted. Turns out it was BPDUs that hadn't been filtered all along. Had they been filtered, we may not have caught the light level problem.

Someone was trying to tell us something. :-)

Anyway, CalREN is clean as a whistle now and should be shifting traffic tonight. I'm going to bed.

CalREN almost migrated in Los Angeles

CalREN engineers shifted their traffic over to the new Internet2 link, and noticed a very small trickle of errors on the port. Light levels are showing up right below the threshold, so we'll need to do a bit of cleaning there. Look for that to happen soon. 

NGIX-W/NREN migrated to Salt Lake City

I migrated the Sunnyvale NGIX-W and NASA connection over to the Salt Lake City router tonight. Always a bit tense when you move that many peers, but this went as smooth as silk. A big thanks to our friends at ESNET who provided a concurrent GigE interface to the Ciena in the Sunnyvale POP so that we could backhaul the circuit to Salt Lake City.

HOPI IP migrated in Los Angeles

We were able to get the HOPI 10GigE interface up in Los Angeles today, across our metro fiber between 600 Wilshire and 818 W. 7th. We tried to get the PAIX 10GigE up to a loop in Sunnyvale, but we're having some metro problems in LA. The signal just ain't getting between the buildings. We have a ticket open to troubleshoot that in the AM. 

LEARN migrated UT Austin behind them on Friday night, so they're now shut down on Houston. That leaves three BGP sessions left on Houston. 

I need to schedule CENIC on LA- the last BGP session on the old Los Angeles router. 

After that, it's just PAIX, NGIX-W and Oregon in Sunnyvale. I'll probably move the NGIX/Ames VLANs this Wednesday, along with PAIX if I can get them up tomorrow. Oregon may be a few more days out than that. 

Pacific Wave, CENIC and LEARN and all the rest...

It's been a month or so. Let me see if I can catch everyone up. We've been mainly busying ourselves with getting some fiber patches and circuits lit on the west coast.

I just migrated all of the Pacific Wave Los Angeles peerings over to the new Los Angeles router. Once we got the link up, it was a painless hot-cut.

We did some metro fiber troubleshooting in Los Angeles this evening and the CENIC/HPR connection is up to the new router. We'll need to coordinate a time to swing their peering over to the new circuit.

LEARN a few weeks ago, and the Texas schools are all working on aggregating behind them. Our piece is done.

USF also looks to be up in Atlanta, so we'll probably be able to shut the old M5 down pretty soon.

We're close on Oregon and PAIX. I'm hoping to get all our pieces worked out this week. There are some interfaces being shuffled around in LA to get them from the old router to the new one. Those should be in place by Friday. We very well could have everyone off the old routers by the middle of next week, if things click. Probably more realistic to expect two more weeks, though.

Houston T640/Ciena installed

The Houston T640 was installed in Houston yesterday and the backbone circuits were lit through the site. Our engineers still have some other work to do today to get the Ciena and PCs installed. Overall, things appear to have gone fairly smoothly.

Denver Abilene router shut down

Denver was one of my favorite routers. Mainly because I love Boulder so much and always got to spend some time out that way. I feel a bit like the delivery doctor and the executioner on this one since I helped install it just under four years ago with Caroline. Debbie Montano stole a few moments away from the Qwest corporate offices to see how things were going. Unfortunately, we gave her a camera and posed for a few photos.

The sad thing is that I have too many pictures of me standing next to network equipment with that same cheesy grin on my face, like I just caught a big bass and am holding it up for the camera. At least Caroline looks like she's putting away some tools or something.

Anyway, Denver hasn't been doing much for anyone lately. All networks were moved off by last week and the backbone links weren't in the path for much of anything. It's been turned down, so our engineers can (gingerly) move it to Houston. We need the Denver power controllers in Houston, so we're going to allow Level3 next week to get them installed for us.

Ah, Past Chris. If Future Chris could tell you anything, I'd probably make sure you know how to lace better. And that you should never look away from your hands to pose for a photograph when you're that close to live DC power. Sigh