Nysernet migrated in Chicago

I worked with Bill Owens this evening to move the Nysernet Chicago peering to the new router. All looks well. Just a few more networks on the old Abilene router to go before we pull it out next week.

Don't feed the engineers

We received a welcome package of frozen ribs, wings and pie in the mail from Rob Vietzke and company from Internet2 last Thursday. Apparently they like the work we're doing. We decided to make an afternoon out of it and had a pitch-in. If you're wondering how this is related to the upgrade, just think about how stable the network was on Friday and what part this lovely surprise played in that. Bravo, Internet2! This was a fun lunch and we managed at least five minutes without talking about the upgrade. :-)

Some brewing discussions

So, it's Joint Techs week, which means you'll see no MAJOR changes to the network. That doesn't mean we're just sitting around watching the snow pile up, though. (Even though I'm told, it's a fascinating sport these days, in New York). We're working on getting BGP up to a few networks this week. Most of them will probably come up without an exchange of routes, just to avoid any routing weirdness for the folks in Minneapolis:

• Nysernet [Chicago]
• Wiscren [Chicago]
• Merit [Chicago]
• Northern Lights [Chicago]

This is a good week to catch up and get a bit further ahead of ourselves in terms of documentation and supply orders. Relative to the pace over the last month, it feels like a bit of a vacation, though. :-)

Nysernet Chicago 10GigE is up

This really capped the day off. Level3 has done an excellent job running down patches between their equipment and Nysernet's local carrier in the past week. The Nysernet interface is up in Chicago, so we can begin their migration.

Internet2 Washington DC node in service

The old Indianapolis T640 has officially been redeployed as the Washington DC Internet2 node at Level3. To facilitate the change, the Internet2 Chicago to New York OC-192 circuit was split in Washington DC and the new router was inserted in the middle. The physical change on the transport equipment took about 30 seconds. The router config took about 30-45 minutes, depending on the service.

MREN transitioned in Chicago

After a few days of wrestling with some faulty jumper cables between the Infinera and MREN Force10 switch at 710 North Lakeshore Drive, I was finally able to transition MREN over to the new router in Level3. Qwest is removing that PIC for us tomorrow and overnighting it to Washington DC for the install crew to place in the router.

Pittsburgh Ciena installed

Tom and John completed the Ciena install in our Pittsburgh node today. It was a bit tight with Tom's plane being cancelled early yesterday, so they didn't get to work until Monday afternoon.

To top things off, Level3 called us late this afternoon to point out that our techs had left a door open at the POP. I got a hold of Tom and John just as they were getting to the airport, so they circled the terminals until I got verification from Level3 on the door and last scanned badge. Our guys turned back to the POP and verified that the door was indeed closed, but likely missed their flight. Someone owes them a few beers.

Starlight 10GigE migrated

Just a quick FYI because I'm dog-tired and sick of sitting in front of a computer today...the Starlight 10GigE VLANs were all migrated over to the new Internet2 router in Chicago. We originally planned to also migrate MREN, but the new circuit is still taking a moderate amount of errors. We believe the issue is isolated to some faulty patch cords at 710 North Lakeshore Drive. Jake Sallee and I will continue to troubleshoot in the morning.

Chicago Metro DWDM up

Jay and I finished the Chicago metro DWDM install around 5PM on Friday. Right now we're transporting the following connections between 710 North Lakeshore Drive and 600 W. Chicago

• T640 to Legacy Abilene Chicago router at Qwest
• MREN 10GigE
• Starlight 10GigE
• WISCnet/Merit Transitional 10GigE

Jake Sallee was a big help in getting all the fibers moved over at 710 North Lakeshore Drive.

We had planned to migrate all the Starlight and MREN VLANs over to the new network, but the MREN interconnect was taking a few errors on Thursday night. That, combined with the Indianapolis T640 router swap-out convinced me to postpone it. We're scheduled to make that move on Monday night.

Philadelphia to New York Dynamic Services OC-192 is up

I received a very welcome phone call from John Graham today informing me that the Philadelphia to New York dynamic services lambda on the Ciena network was turned up and the Philadelphia node was reachable over the in-band management.

Pittsburgh and DC installs next week

Next week will see a few installs in Pittsburgh and Washington DC. John Graham and Tom Johnson will be heading to Pittsburgh on Monday morning (well after the Colts have crushed the Bears in the Super Bowl!) to get the Ciena chassis and support equipment installed.

Following Pittsburgh, John and Tom will head to Washington DC to meet up with Andrew Lee, who will assist on the DC install. The Indianapolis T640 that's being pulled this week will arrive Wednesday morning and we should be able to insert it into the current New York to Chicago backbone circuit on Thursday evening. That will allow the migration of the following connectors:

• NGIX-East
• MAX
• OARNET (from Indianapolis)

Ciena will be arriving in DC and Pittsburgh the following week (2/12) to perform the Test and Turnup of the CoreDirector CI at each site. By week's end, we should have the Ciena backbone up between Pittsburgh and New York. Once that's complete, both PSC and Drexel will be able to migrate to the DC router, as soon as they complete their installs into the I2 suite in Pittsburgh and Philadelphia, respectively.

Indianapolis router swap

With the loaner T640 router that we shipped from New York safely in Indianapolis, it's time to put it back in the network. We'll be swapping it out with the Indianapolis T640 router, so we can send it off to Washington DC. The loaner needs to go back in mid-March, which works well for migrating the networks off. A few of them (the Indiana Gigapop and the University of Louisville) are relying on the DC-Atlanta-Chicago Level3 backbone to be handed over on March 3rd.

Chicago Infinera ring Day 1

Jay and I completed the Infinera install at 710 North Lakeshore Drive, but we have some config work to do in the morning. There was some confusion over the bulk panel from our racks to the fiber meet-me area and I thought we were getting SC-style connectors. Instead, it's LC. We don't have much call for LC-LC connectors, so we don't normally keep them in stock. Jake Sallee has offered to loan us some of their fibers. We'll either swap them out later, or replenish Jake's stock next week.

Tomorrow we're off to 600 W. Chicago to turn up the Infinera ring. We can light one side during the day. The other side currently carries the new Internet2 network to legacy Abilene "network glue", so we've got a maintenance window to work on that in the evening. I'm also planning to migrate the Starlight and MREN VLANs over to the new Internet2 router so we can yank their PICs from the legacy router for use in DC next week.

There was a slight mishap in the POP involving a pile of squishy pink foam and Jay's glasses. The poor man can't see very well tonight, but we have some electrical tape! :-)