Roadmap for the week

We've all recovered from the New York T640 swap and are focusing on the next set of installs. The Juniper loaner chassis we recovered from New York is on its way to Indianapolis where it will be swapped into the Indianapolis T640 on Thursday evening. At the same time, Jay and I will be heading to Chicago this Wednesday morning to install the metro DWDM equipment at 710 North Lakeshore. We expect to be bringing up the following networks while in Chicago:

• MREN
• WISCnet/Merit Transition
• Starlight
• Equinix
• HOPI Internet2 L3
• HOPI L2 backbone to Washington DC

I'm a bit excited to get out of the office. I've been pushing e-mails, spreadsheets, and project plans around so much in the past few weeks, I'm nearing my saturation point. That's always a good time for some DC power exposure. :-)

HOPI New York to Washington DC Circuit

Thanks to the assistance of Chris Tracy and company in Washington DC and Matt Zekauskus who ran jumpers in New York about 2 months ago, the L2 circuit between New York and DC is back in service. This had been blocking on completion of a bulk tie cable between the Internet2 suite and the MAX suite in McLean.

We're going to Chicago next week and hope to get the DC to Chicago circuit up by mid-week.

New York swap complete

The T640 router swap on the 24th floor is complete. Most connectivity was restored at 2215 EST, but the juniper syntax for overwriting a configuration threw me for a loop. It really just merged the two configurations.

The Surfnet OC-48 didn't come back up. It's showing RDI path errors, but we've seen that before during extended outages. The configuration looks good from our side.

The MCIT/ENERGI connection is still unreachable. They've been unreachable ever since we moved to the different T640 interface. I had been operating under the impression that it was a MAC-address problem because it's a TCC connect and the engineer setting it up 2 years ago asked me for the MAC address of our switch. I made sure to have the engineers put the OLD MANLAN 10G interface on the T640 end of the interconnect, but it's still down. Will need to troubleshoot more with the carrier in the morning.

T640 swap commencing

The engineers in Chicago have gone dark as they move to the far side of the moon. We hope to re-establish connectivity soon. Farewell New York engineers and god-speed.

New York Router getting settled in on the 24th floor

It brings me great pleasure to bring you the news that the T640 from the Qwest space on the 14th floor of 32 Avenue of the Americas has made its way up 10 flights of stairs on the backs of Jay Duncan, Andrew Lee and Hans Addleman. The oxygen tanks were empty and the Sherpa made its way back home by the time they reached the "summit".

Seriously, it's been one of the crazier days in my career here at the GRNOC. Not one that I care to repeat, but through the perseverance of some very dedicated people, the router is now sitting next to the Juniper loaner on the 24th floor just waiting to make a nest in the Nysernet space at 8PM EST.

We had a bit of trouble getting the Abilene Washington DC backbone circuit swung up to the 24th floor, but it managed to come up about 10 minutes ago.

Router swap starts in 1 hour and 27 minutes. Stay tuned. We're all heading out to refuel.

HOPI Interface migrated in New York

I migrated the HOPI interface to the new T640 in New York. Took a lot of typing to move around all the MPLS tunnels that were CCC'd across the Chicago and DC HOPI nodes, but I'm pretty sure I got it all.

All that's left on New York is MAGPI. We're aiming to take the old New York router out on Thursday this week. It'll spend about an hour out of the network and take a ride up the freight elevator in 32 Avenue of the Americas to be swapped in place of the current new T640 on the 24th floor. That router is a loaner from Juniper. Rather than leave it in New York, we're going to bring it to Indianapolis where it can help hold things down until a few connectors are ready to migrate off. The timing of the move is somewhat dependent on when we can get the freight elevator operators to work. We'll push it as late as possible.

I'm taking bets on how fast we can swap out the loaner router with the older T640. The power's already there. Turn it down, swap the router out, move interfaces around and replace the config. I'm guessing 20 minutes. :-)

MANLAN migrated in New York

The MANLAN 10GigE circuit has been migrated to the T640 in New York. This was essentially a VLAN move on the MANLAN switch, and Layer3 configuration on the Internet2 T640. All networks are back online except for MCIT/ENERGI, though I can find nothing wrong with the configuration. They've had issues with their network going offline during interruptions of service, so I've asked our NOC to get in touch with them to check their side.

I may still yet tackle the HOPI circuit tonight. Layer2 is already up, but there's a lot of router config. My head has been stuck in transitional issues since noon, so I think I'm going to take a break and watch the Colts like everyone else in Indiana.

MAGPI Layer2 is up

Thanks for some fancy footwork by the MAGPI folks, we were able to get the link up between their router and the New York T640. We're working on configuring v4 BGP/MSDP/IPv6, etc.

MAGPI circuit will need to wait until tomorrow...

The MAGPI wave is still an open issue. The U Penn folks are doing some troubleshooting with their local fiber provider tomorrow morning to find out why light isn't getting to our suite. Apparently there's a splice point through the second floor on its way from the 9th floor to us on the 3rd floor. The provider seems to think that may be where the trouble is. Hope to hear good things tomorrow morning.

I'm tentatively going ahead and scheduling the New York chassis swapout for the end of next week. We've gotta get that going so we can put it in Indianapolis to free that chassis up for Washington DC. Before you know it, Seattle will be affected if we don't get this done. See? :-)

The Ciena of Brotherly Love

I heard good things from Andrew a few hours ago as he was packing up. The Ciena chassis is installed in Philly and all is well. Ciena is currently scheduled to do their test and turnup work the week of January 29th.

MAGPI wave troubleshooting

We're working on getting the MAGPI wave up, but we're not seeing light from MAGPI in Philly. MAGPI is working on contacting their fiber vendors to verify light levels along the path. Hope to get some word soon.

The value of electrical switching on a DWDM system

Thinking back on this, this was a fairly obvious observation, but I thought I'd mention something that I find to be very cool. I knew the theory of why this was so cool, but it was driven home by a real-world example that I ran through early this morning.

Level3 provisioned the MAGPI wave to show up on the second channel in the first bundle of 10 lambdas in New York, but on the first channel in Philadelphia. At first, I was a bit worried because I had some engineers pre-wire this particular lambda to the 3rd channel on the DWDM plan in New York. But then I realized I was thinking like a traditional DWDM groupie from the 90s. So, I took off my flannel shirt, turned down the Soundgarden and really got to thinking from a slightly less flanneled, yet still Pearl Jam'ed 2007 perspective. :-)

In a conventional DWDM system, a signal that's placed on a transponder card (translating your router's ethernet or SONET signal into something that can be multiplexed with other signals) is hard-wired to a particular DWDM lambda. Changing that lambda required changing a transponder card out to a different model- for those systems that were around before tunable DWDM optics- or at least changing the fiber run between the mux-demux card and a tunable card (for those later systems that do have tunable optics) The whole theory of having tunable optics really gets stuck on having to fix that jumper to the mux/demux card and you lose some of the dynamicism that's so heavily touted along with it.

So, back to the Infineras. Since the signal is electrically regenerated at each node, switching the lambdas around (and switching the client signals around) is a piece of cake. On the New York Internet2-owned DWDM ring, I have a client signal coming into port 2 from Level3 at 111 8th Avenue. I wired the 32 Avenue of the Americas node's client interface up to port 3. Instead of having to re-wire the 32AoA side to dump the signal out on port 2, I can simply create the circuit between ports 2 and ports 3 and be done with it. There's essentially a complete decoupling of port 2 and channel 2. The two have nothing to do with each other.

I knew you could do this on the lambdas that crossed the backplane between different fiber directions, and I suppose I figured I would be able to do it within a particular card, but it just dawned on me just how powerful that feature really is from a real-world operational standpoint. This saves us from having to move jumpers and allows us to bring circuits up really fast. Theory is great, but when you start using it, the value just pops off the screen.

Philly update

Andrew and John report that things are moving along in Philadelphia. There's an odd problem with the rack not being quite wide enough and throwing off some of our rack ear extenders. We're overnighting some 5RU extenders, which is unfortunate because they'll hang into dead space and make it difficult to put any future 23" equipment there. (But let's be honest, whatever goes in next will likely be 19" anyway). They were also missing clear heat-shrink from their kit of materials, so that got overnighted as well.

I'm working with Level3 on getting FDP assignments for the lambdas coming out of Philly (and actually turning up the MAGPI circuit that's been ready to go since Tuesday). Hope to have that info by tomorrow morning so they can run patches and roll the fibers if need be.

Philly Ciena install underway

Andrew and John are relaxing comfortably in a Northwest Airlines coach seat right now, dreaming of the Philadelphia install that will happen in a few hours. They should be on-site by 11AM and start knocking through the install shortly thereafter. Unfortunately, with these installs, there's no "EUREKA!!" moment when the network comes up since we're having Ciena slot the cards and turn up the node. In general the techs will have a POTS line to install, but who gets excited about modems anymore?

We're hoping to get Ciena out within the next few weeks to complete their part.

Philly rack swap complete

The Philadelphia delays are even more slight than I thought. There was an e-mail in my inbox yesterday that I hadn't gotten to yet. The Philly techs replaced the rack immediately with the appropriate 23" rack and mounted all our hardware. The power, bulk panels and multifunction panel are all installed. It's ready for us to visit and get the Ciena installed.

Slight setback in Philly

I got a rather confusing call from a Level3 POP technician in Philadelphia today. He was installing the Sentry power controllers and said he didn't have the right rack years. The controllers ship with 19" ears, but we bought a bunch of 23" versions. I was certain that we included the 23" ears in that shipment. It turns out that the rack delivered in Philadelphia was a 19" rack instead of 23". That isn't going to hold the Ciena, so Level3 is swapping it out. My understanding is that this should happen before this Friday, but it's tightening up the MFP and power controller install. Assuming it happens within that timeframe, no harm, no foul.

Ciena slotting and On-Center

After a bit more discussion with Ciena, we've decided not to pre-slot the CoreDirector CI cards before we leave the install. The Ciena Test & Turnup technicians normally handle that and it's best to leave it to them. They'll be checking for bent pins and those sorts of hardware defects, so they'd want to remove the cards anyway. We'll just leave dangling fibers (*gasp*) and they'll route them for us.

Word came down from our system engineering team that the Ciena EMS was successfully installed on our management box. The software isn't the most intuitive piece of equipment to install and Ciena was kind enough to arrange for a technician to come out and help out twiddle the appropriate bits on the system to make it all gel. Both New York and Chicago and already visible.

BTW, Ciena has officially certified both Chicago and New York, as of yesterday. We have a bit of test data to look over, but that will be more of a formality.

Eat more fiber

There are a lot of questions about all the different fiber panels on the new network. In general, there are three different types of panels in use:

Bulk panel to Level3
These panels terminate the bulk fiber between the Internet2 suite and the Level3 centralized fiber termination block, known as the OSX. This bulk cable is used to carry client signals between equipment in our suite and the tributary cards on the Infinera transport gear. In router locations we have 96-strands (48-pair) of fiber going in each direction out of the city. So, in a three-way router site, we have three 96 strand cables going to the Level3 OSX. At smaller Ciena-only sites, we have 48 strands (24 pair) going in each direction.

Each 96 or 48 strand bulk cable is terminated across 4 or 2 single RU ADC fiber panels. These panels terminate 24 strands of the cable into SC-connectors. The panels are divided into two sides, with the left side of the panel aiming fibers out the left side of the rack, and the right side aiming fibers out the right side of the rack. (This becomes important when determining fiber lengths.) There are two rows of 6 strands on each side of the FDP. Strands 1-12 are on the left side and 13-24 are on the right side. The ports are numbered from left to right, with 1-6 on the top of the left side and 7-12 on the bottom of the left side. Level3's standards are to use ports that are on top of each other, rather than sequential numbered ports that are side by side. Hence, their circuits will terminate on ports 1 and 7, 2 and 8, 3 and 9, and so on. The right side of the FDP is numbered similarly, with the first port being 13.

Multifunction Panel
These panels are very simplistic 1 or 2 RU panels that have modular-style connectors. In router sites where we're obtaining a Level3 High-Speed IP (HSIP) ethernet drop or a coaxial timing connection, we have modules to support that. The rest of the connectors are generally SC-style to terminate customer fibers that land in our suite. We've issued LOAs to customers to allow them to land fibers within our suite. No customer fiber is to connect directly into our gear, aside from those within our suites. The customer fiber should land in the MFP on the back of the panel in the designated ports. Internet2 is responsible for the fiber patch between the MFP and the various pieces of equipment within the suite.

Ciena Fiber Trays
The Ciena interfaces often require simplex fibers to be run out different sides of the card. It's acceptable to run simplex fibers within a particular rack, so at Ciena-only sites where the Bulk Panels to Level3 are within the same rack, it's an acceptable fiber run. For router sites where the bulk FDPs are often in another rack, we decided against running the more fragile and less-available simplex fibers through the overhead fiber infrastructure and are placing a 1RU fiber distribution tray on top of the Ciena. This tray converts the LC-style connectors on the Ciena to SC-style, so we can maintain stock of the more common duplex SC-SC fibers.

Miscellaneous FDPs
There are some additional panels that are used in some of the 3rd party sites that fall outside of the norm. They are:

• NEWY32AOA - bulk FDP to Fiber Meet Me Room (FMMR) - 48 strands
• CHIC710NLSD - bulk FDP to 710NLSD Fiber Meet Me Area
• WASH - bulk FDP to MAX suite on the same floor - 72 strands
  

Ciena turnup

We've been discussing the Ciena node turnups a bit lately, with the majority of the coaching coming from John Graham. Recall that there are two types of sites that have a Ciena in them. Router-node sites, where the Ciena install will be timed to coincide with the router install, and Ciena-only sites that usually just have a single rack of equipment. Each turnup has three main phases

Installation
This is done by an Indiana University install team of at least two people. It will include getting the chassis into the racks, and terminating the power on the Ciena. Getting the chassis in the racks at a remote site can be challenging, given the weight issues. The Level3 standard racks have a bit of extra space at the bottom, between the lowest RU and the kickplate. That likely means that the screw holes won't line up properly with the Ciena rack ears. Getting those to go up with a two-man team can be a bit dodgy at times, so we're looking into some sort of front and back jack system that we can ship with the install kits to give that small 1/8" bump that it's hard to do without the Incredible Hulk as part of the install team.

Engineers will install the observatory and control PCs, the small racklan switch and the out-of-band dial-up router at this time, but there will be no data plane connectivity because we aren't turning the Ciena up. We'll pre-run the backbone fibers from the Level3 bulk FDP to the Ciena, but they will be clipped to the 10G cards and not inserted.

Test and Turnup
Turnup and configuration of the node is left to Ciena. They will arrive shortly after we've handled the chassis install and power and turnup all the cards. After each card is slotted, they'll run them through a series of traffic tests that would make even the most cautious of us feel somewhat like a network cowboy. Ciena has agreed to have their installers plug the pre-run backbone fibers into the OC-192 backbone cards when they've completed testing.

Node Certification
Once the card is installed, and an adjacency has been established between the new Ciena and an already-activated Ciena, it will become our playland. But first, John needs to do some quick checks of the system to make sure it's good to handle production traffic. A lot of that also has to do with documentation and monitoring. We don't want to put anyone on the Cienas without proper back-end support.

And there you have it. Ideally this will be a two-week process (one week spent getting it installed- only one day of which will actually be used to handle the install) with the next week for the test and turnup.

Site-by-site update

Things have been moving so quickly this past week that I haven't had much time to sit down and type up an update. Apologies all around and I promise life will get better!

Chicago
We initially thought we'd get the DWDM ring installed in Chicago this week. The power work was complete last week, but the bulk fiber panel from our rack to the 710 North Lakeshore Fiber Meet Me Area didn't make it in time. The last estimate I have is a week or two, but we're working to get that tightened up. We have the option of doing temporary long jumpers, but in our business, temporary things tend to go undocumented and become a part of the permanent landscape. We'll avoid it as long as we can. It won't be a picnic to not have that up until the end of January, but it won't really push things out too much since one of the connectors won't be ready to move off the old Chicago router until the end of the month. Hopefully we can get that date pulled in.

Philadelphia
The suite was accepted and the power installs are on track to complete this Thursday. The MFP is also being installed by Thursday and we expect the MAGPI cross-connect from their metro dark fiber provider to be terminated on that panel on Friday. That leads us to New York....

New York
Once MAGPI is out of New York, we can remove that router from the Qwest floor and move it up to the new space. We'll swap out that Juniper loaner that's been handling some of the new connections already and ship it to Indianapolis to pull its weight for us there.

After a bit of MTU-wrangling, the new NOX R&E peering is fully transitioned over to the new router and they're going to turn down their OC-48 on Thursday.

I talked with Jim Shaffer, who's in New York this week helping with the ESnet install at 32AoA. Things went well with ESnet, apparently. They sound like they've accomplished their install in two days time. Kudos to ESnet and welcome to the new Internet2 suites. I wish I left a doormat for you....

Pittsburgh
We got back the photos for our remote suite acceptance and found enough issues to reject it for now. There's a call tomorrow with Level3 to go over the bits that we think need addressing and hopefully it'll get taken care of within the next few weeks. We do have Cienas to install, after all

Cleveland
We heard last Friday that Cleveland was actually completed on December 21st. After looking at accepting the suite remotely, we decided it would make more sense to do it in person. We're accepting all the router sites in person. There are two sites (Cleveland and Sunnyvale) that are multi-row Ciena-only sites. It's difficult to check things like inter-row signalling trays and other overhead infrastructure, so we'll be doing those two sites in person. Jay Duncan is enroute and will be on-site tomorrow afternoon to handle install.

Boston
It looks like Level3 has located a suite that recently came available to us, so we're going to go out on our own instead of colocate in the NOX suite. A big thanks to Leo Donnely for all his work in entertaining that option.

Back from the holidays

We're still all recovering a bit from the past few weeks of mismatched work schedules, but people are trickling back into the office and wiping the end-of-the-year sleep out of their eyes.

Today was spent getting people back up to speed. A few highlights.

• We're awaiting some metrics on the suite in Philadelphia so we can accept it.
  
• We were notified that the Pittsburgh suite was completed today
• Once we get Philly and Pittsburgh accepted, we'll get power and multifunction panels installed within a few days
• MAGPI is close to being ready in Philadelphia. They need to procure one last jumper from their metro provider to our suite. Hope to get that done in the next week
• NOX is going to switch over their primary traffic to their new 10G on Thursday and do a 1 week burn-in
• The new Internet2 NOC website address is ready and a lot of the text has been changed to the new name. (But I won't tell you the address just yet!)
• Ciena has shipped the Cleveland, Philadelphia and Pittsburgh chassis. We'll install shortly after the DC power is installed by Level3. Ciena is waiting on a new revision of their OS to hand over the devices in Chicago and New York.
  
• The HOPI lambda between New York and DC is blocking on re-termination of the bulk cable between the I2 suite and MAX's suite in Washington DC.
  
• Jake Sallee tells me that power should be ready at 710 North Lakeshore Dr. in Chicago by tomorrow. That likely means the DWDM metro gear will get installed next week.
  

There have been a few scenarios floated around to get a router to Washington DC. It's almost certainly going to be the Indianapolis T640, but we need to get the DC router installed before the Atlanta to Chicago Dedicated Wave System (DWS) becomes available. So, we can either move people off Indianapolis ahead of that, or we can put a new router in Indianapolis. The current thought is to get peers off New York within the next few weeks and move that Juniper loaner router to Indianapolis. We should know more in the next day or two whether that's going to be possible.

As Rob Vietzke said this morning, "it's time for the rubber to hit the pavement."