Chicago Install Day 4

The new Chicago T640 is up and connected to the Chicago Abilene router. First, a little background on a theorem first postulated by noted scientist and GRNOC engineer, Caroline Carver. It states:

Actual Install Time + Time Exerted By External Forces = Time Alloted for Install

Put differently, no matter how much extra time you put into your schedule, the install will grow to fill that time. Chicago was no different. We got the metro fiber up 2 minutes before noon, the time we set to leave. Q.E.D.

We still have some open issues to deal with when we return:

• I hate the way we left the AC power cables. We'll reterminate them when we're back in town for the I2MM in early December.
  
• We still need to get the POTS line up, but we can do that remotely. Should be in next week.
• We need to swap out the 16 port asynch WIC card on the 2611 console server with the 32 port card that shipped after we left for the install
• We need to swap out the LR optics between the HP switch and the T640 with the SR optics that shipped after we left for the install
• Power up the Infinera
  
• The observatory machines need to be given their IP addresses
• Need to load SSH code on the Cisco 2611
• Finish configuring the T640
• Document everything in the database
• Get monitoring up

I'm going to do as much of this as I can this weekend since we're tight on time next weekend for the New York install. Given the amount of time spent in Chicago without the Infinera metro turnup, we'll need to work faster in New York next week when we do have an Infinera ring to turn up.

As for the route engine, I think it was a problem with the code not being at the same level as the primary RE and having a config that was out of synch. We got them both working before we left and rebooted it a few times to be sure. I moved the code down to JunOS 7.4R3 to bring it in synch with the production routers that were also downgraded last night to get around a fairly significant Juniper bug that keeps resetting our FPCs.

A big thanks to the POP technicians at Level3. Ryan and Chris did a great job working through the metro fiber mixup (in addition to the folks behind the scenes). Hope we don't run into it again, but in this case, things worked out just in time.