Wednesday, November 29, 2006

NEWY and CHIC production updates

I pushed the IPv4 and IPv6 unicast and multicast configuration out to all the legacy Abilene routers today for all the current and future upgrade routers. That brought IPv6 up to the new network.

Chris Small and Luke Fowler made great strides in getting the two cities up to snuff from a monitoring perspective. The NOC has them in all their tools. We monitor backbone IS-IS adjacencies and BGP neighbor status at a one-minute resolution. The routers are in the router proxy.

This enabled Matt Davy and I each to work on getting peers up on the New York router. Nysernet's BGP IPv4 BGP session is up (the first US connector, yay!), and the NOX underpinnings are complete. These are new connections that are running in concert with their old New York router connections, so we can ensure things look kosher before turning down old circuits.

The Ciena install teams are making good strides in both Chicago and New York. They should be done with their work by Friday.

Rob Vietzke will be back in New York tomorrow to do some testing of the iHDTV equipment for a plenary HD event. While he's in town, he'll be running some additional patches from the Level3 bulk panel to our Infinera at 111 8th Avenue to help support some other demos. Hopefully those fibers arrive in time.

Our POTS lines in both Chicago and New York are delivered and are ready to be plugged in.

Tuesday, November 28, 2006

NOX is up/up in New York

The NOX temporary wave to New York is up/up in New York. Matt Davy will be working on their interface configuration and they should be passing traffic shortly.

Monday, November 27, 2006

Kansas City and DNS

Tom Johnson was in Kansas City today to walkthrough the suite. Looked good except for a few missing details. We'll have to follow up with Level3 on each of those.

Rob Vietzke's video work in New York went off without the HDTV encoders, so he'll need to return on Thursday. He pulled back some Ethernet cable for the Ciena installers to use and noticed that Verizon had delivered our phone line. Apparently it went to the wrong rack and he didn't have an RJ-11 coupler with him, so we'll get that plugged in on Thursday.

Caren got colo worksheets into Level3 for Raleigh, Nashville, Louisville, Indianapolis and Atlanta. We're going to be in a bit of a crunch on those.

We figured out the DNS question. Looks like we're going with net.internet2.edu. The new NOC account will be noc@net.internet2.edu. I2 is sub-delegating that zone to us so we can auto-generate our DNS off our database. Luke Fowler is already starting to work that up.

I worked on the temporary NOX circuit over the Level3 transport network between Boston and New York. At the end of the day, we were able to get light on the New York side, but not on the Boston side. Brian Mollo from NOX confirmed at their rack that there's a missing jumper from Level3. We'll try and get that wrapped up tomorrow.

We're having some very odd problems on the HOPI Chicago management racklan. Devices keep disappearing from the network, but only from certain points of view. Like, we can ping a device from IU, but not from the racklan itself. We can ping devices from some racklan devices, but not from others. They're all in the same subnet and the arp tables look good. It's almost like there's some hidden layer2 ghost in the machine. I mention it because our Chicago Abilene interconnection runs through the HOPI Force10- which is currently unmanageable. Andrew and I looked at it this afternoon and can't come up with a reasonable idea where the problem is. We clear the arp table on the switch and some things come back up...sometimes. Sometimes they don't. And they don't come back up from all points in the world either. Some devices just reachable on the racklan, some just reachable from off the racklan. Clear the table again, and the device reachability reverses. VERY strange. Need to look at it in the morning with a clear set of eyes.

Saturday, November 25, 2006

Roadmap for the next week....

One week left before the I2MM and lots to do. A preview of the hot-button items this week:

NOX 10GigE to New York

Level3 delivered it on Wednesday after everyone had left for the holiday. We're good to go in New York. I have it running to a loop from 32AofA to 111 8th Avenue. No light from L3 yet, but I need to call it in first. I'll do that on Monday.

I2MM HDTV Demo Testing
Rob Vietzke and a small crew will be in New York on Monday to get the logistics worked out for a video conference to take place during the I2MM plenary over the new network. Gotta make sure that moves smoothly

Ciena Installs
Ciena will be doing the CoreDirector CD install and turnup in Chicago and the Test and Turnup of the chassis we installed in New York last week. Not sure if we'll have the OC-192s up between them before the I2MM, but there's currently nothing slated for them...yet.

Chicago 710NLSD Install Trip Planning
Since we'll be in Chicago for the I2MM the week of December 4th, we'll want to finish up the metro ring install, assuming the 710NLSD site is ready. Need to work with Jake Sallee on that to make sure we're in good shape for a Monday or Tuesday install. Hell, if the power can be ready by Friday, we'll try and go up early to get it up before the I2MM.

Colocation Worksheets
We've gotta work like gangbusters to get the colo worksheets done for the rest of the sites on Ring 2. Caren's got Indianapolis in and we need to try and get as many as possible in by COB Friday.

New York and Chicago router monitoring
The only thing holding me back from migrating circuits in New York is the monitoring. Hope to get that all worked out by Tuesday or Wednesday. By I2MM time, we should have several networks passing traffic on the new node!

Hope everyone had a restful Turkey day. Back to work on Monday!

Tuesday, November 21, 2006

A gaggle of connections in New York

Matt Zekauskus was gracious enough to provide some remote hands support in New York for us today.He took care of a few housecleaning tasks and installed some hardware and jumpers to support the following new connections:
  • Nysernet - the first connector on the upgraded network
  • NOX - should be up tomorrow on a temporary wave from Level3
  • MANLAN - not production, but ready to be used to transition MANLAN networks shortly
He also fixed the missing default route on the OOB router at 111 8th Avenue, so we have console access at that location.

We still need to do a return trip to get a few items taken care of. The power inverter for the Ciena management servers isn't up ready yet, and all the AC power cables need to be cut to length and re-terminated. There's a whole bunch of documentation that we decided to handle remotely, but there will likely be a few copper cables that will need to be traced by hand.

All in all, things are shaping up nicely and we'll be more than ready to go for the I2 Member meeting on December 4th.

Monday, November 20, 2006

First backbone link is UP!

We just hit a major milestone. From the Chicago router:

chrobb@CHIC-re0> show interfaces so-0/2/0
Physical interface: so-0/2/0, Enabled, Physical link is Up
Interface index: 147, SNMP ifIndex: 45

Received path trace: NEWY-re0 so-0/0/0
4e 45 57 59 2d 72 65 30 20 73 6f 2d 30 2f 30 2f NEWY-re0 so-0/0/
30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0...............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 0d 0a ................

Sunday, November 19, 2006

Infinera OSC data plane

Infinera confirmed that the auxiliary port on the front of the chassis is indeed a routed connection and not a Layer2 passthrough. That explains why I can't ping through to the 111 8th Avenue racklan from the 32 AofA racklan. I'm going to need to re-address the machines in that lan into a distinct subnet. No big deal, but I wish we'd had time to test this before we deployed.

Of course, this would be completely trivial if I hadn't spaced the default route on the console server at 111 8th Ave. Tom was doing the final clean-up work for us when the Level3 techs delivered the 2MB ethernet high-speed IP drop from their commodity network. (We're using those in the router cities as a high-speed backup) He asked me if I wanted a default route toward Level3 and I just wasn't thinking at the time and said "no". Realized after we got back, that it's needed since there's no BGP with L3 over that link. It's on the list of things to fix when we head back. *sigh* It'll take 2 seconds. :-)

Friday, November 17, 2006

New York Day 4




Writing from La Guardia....

We started the day out on a down-note with Aaron feeling a bit under the weather. So, we started out one man short. Infinera called me last night about the OSC channel and we agreed to start troubleshooting around 10:30 AM. That meant we had to get the Abilene interconnect to the loaner T640 chassis up by then. Worked on the config last night and this morning.

In order to connect the new network with the old network, we're re-using a pair of Internet2 owned fiber from the 24th floor (new node) to the 14th floor (Abilene node). In order to do that, we have to move an existing Abilene connection to the new router to free up that inter-floor fiber. We chose the OC-192 interconnect between the MANLAN Nortel HDXc and the Abilene T640. That circuit has an OC-48 carved out of it for Surfnet that doesn't currently carry any traffic. Lowest impact. Lowest problem.

We decided to frame the inter-floor fiber as a 10GigE so we had enough OC-192 interfaces to the new Level3 backbone circuit. Tom played remote hands down at the Qwest POP to replace the PIC in the Abilene router. Couldn't get it to come up. Tried reseating and resetting the PIC. Nothing. Good light, just no link. Eventually, I moved the circuit to a different 10G interface and it greened right up. Bad XENPAK. Well, not bad, just not a Juniper XENPAK. Came from another vendor.......

Once we had that interface up, we tried getting the Surfnet circuit to come back up. No amount of config would make it happen. Eventually we saw the SONET circuit throwing RDI-P (Remote Defect Path) errors, which would indicate a problem on the Amsterdam side of the circuit. Talked with Caroline to see if she could follow up with them. Ball's out of our court. Moving along.

And move along we did. 20 minutes to get the router up to production speed and make sure I could route through it to the rest of the R&E world. DONE! Infinera calling!

We did a bit of work on the Infinera to verify that I configured everything correctly. Of course (*grin*) I did and all was well. We didn't get the IP connectivity via the OSC channel layer2 payload working, which means that I can't see the racklan at 111 N. Canal from 32 AofA. Since we could manage both sides, I decided to move on and work on the IP connectivity from Indiana on Monday. We configured an OC-192 up to a loop at the 32AofA side and I was happy. Done with that.

By then, Aaron started feeling better. Tom had been working on Ciena power and just finished. So we moved on to housecleaning items. Tom went over to 111 8th Ave. to plug a client fiber in for the new backbone circuit and run a copper cross-connect to the Level3 high speed IP drop that L3 was installing into our MFP. He got that interface up. We got everything documented and all is well.

We'll likely go back in the near future to clean up the racks a bit. We didn't get the Ciena control PC inverter going and we didn't get a chance to cut all the power cables and ethernet cables to length.

A VERY big shout out to Jim Shaffer for all his work. He did a great job staying on the contractors about the power install issues and making sure we had what we needed. I'm sorry to say we were a lug or two short, and he helped us out. Good support at the POP can make an install go so much more smoothly. I'd hate to think how this would have gone without him around.

Photos are updated. We're going to enjoy a nice quiet weekend before we get to deal with this again next week. :-)

New York Day 3

The house power at 32AofA was completed today, though it was fun bouncing around in the small space between us, the electricians and Rob Vietzke. At the end of the day, the T640 is powered up, the Ciena is racked and grounded, the Infinera is turned up with light between sites, and most of the PCs are up and running.

We had some challenges with the metro fiber when we noticed a pair of fiber in the two ports that Level3 gave us in the LOA in the FMMR. Had to get something from them in writing before we yanked it since they don't run fibers in the FMMR. Miraculously, the two sides saw light from each other. I have the OSC channel up and can manage a remote node through the local node via the embedded Infinera management software (a Java applet), but I can't ping between the two Infineras or ping through the Infineras to the two racklans that are bridged across the OSC payload. That's going to be a problem since the 1118th Ave site is getting it's racklan connectivity across that OSC to 32 AofA. Confused? How about a diagram?


Tom had all sorts of trouble with our power inverter. It wasn't well documented and the plastic rails that hold the modules in kept falling out. Clearance on the lugs was minimal. It's a positive DC system, so we had to wire it backwards. AC voltage fluctuates when you put it on the standby control card. I don't think we'll be using them anymore. We had to in New York since there's no house AC service at 32 Avenue of the Americas.

We've got a LOT to get done tomorrow. Have to configure the T640 early on so we can stick it in between the Abilene router and the MANLAN HDXc OC-192. Loads of fun.

Thursday, November 16, 2006

Light from 111 8th Ave

Well, the metro fiber issue appears to be worked out. I turned up the Infinera node at 111 8th Ave about an hour ago and we're seeing light at 32AofA. I'm back in the 32AofA space now, working on the other node. Fingers crossed.

The Ciena showed up, too. Lots of work to do tonight....

New York Day 2

Started the day off at 32 AofA this morning. The observatory rack has power to it, but the electricians ran out of cable for the T640 rack. We did a little work and headed back over to 111 8th Ave to tie up some loose ends. We noticed yesterday that our multifunction panel was mounted in the wrong rack, so we moved it over. That required finding an 8-foot ladder in Manhattan. Then, once you get a ladder, you have to go in the loading dock, which requires some amount of clearance or social engineering of the dock manager. Lots of fun!

The rest of the day was spent back at 32AofA mounting our equipment and getting the power as done as we could get it without house power.

We're having a bit of trouble locating one side of the Level3 metro fiber in the 32AofA FMMR. Level is going to send a tech out in the morning. After that, we can work on getting the Infinera ring up.

Here are some photos.

Wednesday, November 15, 2006

New York Day 1

I didn't get an update out last night because the hotel stuck me in the farthest corner of the building where I couldn't get wireless. Oddly enough, my cell phone's GPRS connection wouldn't work either....

Got into New York around 9:30. Dropped our stuff off at the hotel and went to 32AofA. We immediately noticed that the power wasn't complete. There was a mixup with the power contractors doing the ESNET install first. We weren't planning on doing anything at 32AofA anyway, so no harm. Jim Shaffer from Nysernet got on it and it's being worked on.

We got everything installed at 111 8th Avenue, though we're missing some fiber jumpers that haven't arrived yet. Jay is overnighting them, so they should be here Wednesday morning. One issue is the MFP panel that Level3 mounted is flush against the bottom of the DC sentry. Doesn't harm us in terms of mounting equipment, but it won't work for running new connections into the back of it. The live DC cables hang down into the space, so I'm certain L3 won't want to work around it. We'll need to get that moved. To do that, we'll need to find a ladder so we can get up to the fiber trays. Jim Shaffer thinks there's a hardware store near 111 8th Ave. Need to figure out how to get that into the building since most places have dock restrictions here in New York.


More when we get to 32AofA....

Monday, November 13, 2006

MREN I2MM connection up in Chicago

The folks at 710 North Lakeshore replaced the pluggable optics on their end and the interface between the Chicago T640 and MREN came up. Excellent work!

Sunday, November 12, 2006

The birth of a new backbone!

Everyone in the R&E community should now be getting the new Internet2 network backbone advertisement. And it goes a little something like this, hit it!

chrobb@InGigUL-re0> show route 64.57.16.0

inet.0: 204220 destinations, 384262 routes (204212 active, 6 holddown, 8092 hidden)
+ = Active Route, - = Last Active, * = Both

64.57.16.0/20 *[BGP/170] 00:11:09, MED 0, localpref 300
AS path: 11537 I
> to 192.12.206.249 via ge-0/2/0.10


This will become part of our standard techno vocabulary very soon, but I don't have enough room in my head right now to remember it. Thank god someone invented computers to have a place for me to store my network advertisement reminders.

No DNS on this yet. Need to start discussing how to handle that. There are some that want everything to be "internet2.edu", but there are some technical hurdles there. Hope to have this worked out this week so I can go back to forgetting what the network advertisement is. :-)

Saturday, November 11, 2006

Chicago Install Day 4

The new Chicago T640 is up and connected to the Chicago Abilene router. First, a little background on a theorem first postulated by noted scientist and GRNOC engineer, Caroline Carver. It states:

Actual Install Time + Time Exerted By External Forces = Time Alloted for Install

Put differently, no matter how much extra time you put into your schedule, the install will grow to fill that time. Chicago was no different. We got the metro fiber up 2 minutes before noon, the time we set to leave. Q.E.D.

We still have some open issues to deal with when we return:
  • I hate the way we left the AC power cables. We'll reterminate them when we're back in town for the I2MM in early December.
  • We still need to get the POTS line up, but we can do that remotely. Should be in next week.
  • We need to swap out the 16 port asynch WIC card on the 2611 console server with the 32 port card that shipped after we left for the install
  • We need to swap out the LR optics between the HP switch and the T640 with the SR optics that shipped after we left for the install
  • Power up the Infinera
  • The observatory machines need to be given their IP addresses
  • Need to load SSH code on the Cisco 2611
  • Finish configuring the T640
  • Document everything in the database
  • Get monitoring up
I'm going to do as much of this as I can this weekend since we're tight on time next weekend for the New York install. Given the amount of time spent in Chicago without the Infinera metro turnup, we'll need to work faster in New York next week when we do have an Infinera ring to turn up.

As for the route engine, I think it was a problem with the code not being at the same level as the primary RE and having a config that was out of synch. We got them both working before we left and rebooted it a few times to be sure. I moved the code down to JunOS 7.4R3 to bring it in synch with the production routers that were also downgraded last night to get around a fairly significant Juniper bug that keeps resetting our FPCs.

A big thanks to the POP technicians at Level3. Ryan and Chris did a great job working through the metro fiber mixup (in addition to the folks behind the scenes). Hope we don't run into it again, but in this case, things worked out just in time.

Friday, November 10, 2006

Chicago Install Day 3

We ended up going back to the hotel and its high-speed internet connection to pull down the Juniper code. After a bit of lunch, we returned to the POP to finish up our work. Had some issues loading the code with the on-board flash of the T640 route engine being half the size of our other route engine. The router kept complaining about being out of space. Found an extraneous file and deleted it. During troubleshooting, I noticed that the second route engine on the chassis wasn't fully seated. When I inserted it and rebooted, the router came up in a very scary ROMMON or bootstrap mode. After a few hellish minutes where I thought I had somehow hosed the primary RE hard drive by deleting that file, I realized there was probably some reason why that RE had been removed in the first place. When I took it back out, the chassis came up just fine. Got the code loaded and all the FPCs are recognized. Need to get a case open with Juniper on that RE first thing tomorrow morning.

We got all the physical work completed and most of the software config in place. The only piece missing is the metro fiber. Level3's been working hard at this behind the scenes, but we're getting nervous since we have a hard stop at noon tomorrow. I'll be surprised if we can get the interfaces over the metro fiber up before leaving. That may mean a separate trip back if we need to do any serious debugging on the fiber with Level3 remote hands.

BTW, we picked up chairs from Target on our first day. Best purchase ever. We'll definitely be putting those in every POP.

For now, I'm happy that Jay and I both got a few slices of Chicago pizza and have a horizontal bed to sleep in.

Thursday, November 09, 2006

mid Day 3 update

Spending a bit of time connected to the Internet via my cell phone's GPRS connection because.....

The North Carolina T640 shipped with JunOS 7.1. The two FPC3s we're using for the 10G connections came from a pool of new FPCs that I2 purchased from Juniper just recently. So, they're of the "Enhanced" variety, which is a newer revision of the linecard hardware. Those newer linecards aren't supported by the relatively older JunOS 7.1 code, so we can't bring up the FPC3s. We need the FPC3s to power the 10G ethernet card that will connect us over to the legacy Abilene backbone. But we need that backbone connection up to get the JUNOS code to support the FPC.

So, I'm downloading the code over my cell phone connection right now, but I suspect we'll end up in a coffee shop within the next hour at the rate it's going. In fact, this posting is taking up precious bandwidth, so I'll stop now.

Chicago Install Day 2

We got off to a late start on Day 2. The badge reader was still not working for any of our Level3 badges. It took a while to get someone out to the POP that could let us in, so we lost about an hour and a half. The POP tech that was there spent a good part of the day working on that reader and was able to get it working for us in the afternoon. It wasn't reading his badge either, so at least we weren't unique. We also got our building badges, so we can get through the lobby more easily on Day 3.

We got everything mounted into the racks and most of the power run. The AC power cables gave us some trouble since they're three times as long as what we need. All our PCs in the old POPs were DC powered, so the power runs were of custom length. A bundle of 12 long AC cords is pretty heavy, so we need to figure out a way to give it enough support to not yank the plugs out at the power controller. I think we'll ultimately have to cut these cables to custom length or try to order some custom ones, but we can't continue to use 10 foot AC cables for 2 foot cable runs.

We got the DC power run to the T640. The Infinera runs should be easy to get done in the morning since you don't need to lay on your back to do them (sorry, Jay!).

Level3 is working on the metro fiber jumper runs and the POTS line. We should be able to wrap most of this up on Day 3.

I placed up a gallery of images from the install process. Enjoy!
Chicago Install Photos

Wednesday, November 08, 2006

Chicago Install Day 1

Boy would I like to go to bed...

We got off to a bit of a late start and arrive in Chicago around noon. Had some time to kill before our Level3 badge session, so we spent a few minutes at 710 North Lakeshore Drive talking to Jake Sallee about colo requirements. It looks like the old racks that are in our space haven't been removed yet, but they should go any day. He's going to get me the exact number of usable RU in their 23" 2-post racks. They do use 1RU of space for a 48-strand LC bulk cable to their fiber aggregation area, so we need to plan on that.

Went over to Level3 to get building badges for 600 W. Chicago. Longtime readers will remember that we were told Level3 supplies building badges to get us through the lobby at 600 W. Chicago, so that's what we asked for. There was a bit of confusion on what we were really wanting and the badges weren't ready. They apparently take 24 hours to activate. So, while we could theoretically get into the Level3 suite in the building, we won't make it past the lobby security to get to the suite. Level3 was able to give me a single building ID to use as a temporary pass until the permanent ones are activated.

I said "theoretically" get into the suite because the Level3 site requires badge-specific access PIN numbers on the colo area keypad. A palm scan is needed at 111 N. Canal and a user-defined PIN is needed at 600 W. Chicago. Unfortunately, no one gave us those PINs. The POP tech that was at 600 W Chicago generated them and tried getting them into the system, but he had to go before we could verify. So we were left keeping one man inside the POP to open the door until the Level3 techs over at 111 N. Canal could get the badge reader to start accepting our new PINs. Of course, the area outside the colo area is lead-lined, so our phones were useless. Had to go out to the loading dock, which meant a lot of back and forth when they wanted us to try something. Eventually Level3 discovered that the badge reader wasn't communicating with their system properly and they were sending a tech out. No worries, we can continue to work inside.

The power run was completed and the Level3 electricians did an excellent job terminating the feeds into our Sentry controllers. They weren't powered on at the BDFB, like we requested since the POP manager wants to see our internal rack wiring first. Every POP is a bit different in that regard. It's fairly rare that we're told that a review of our wiring is needed in advance of power install. I'm sure he'll scowl at the all-black battery and return cables that Jay brought along, but we'll have to cross that bridge tomorrow. (All black cable is much easier to transport since you don't have to worry so much about lengths of each and running out of one before the other- yes, it DOES happen!)

We got the Juniper T640 racked up with a bit of trouble. In the old Qwest POPS, we were fortunate enough that the 2-post rack hole spacing aligned perfectly with the hole spacing on the T640 ears when the T640 was resting in the bottom of the rack on the kickplate. Not so with the Level3 racks, unfortunately. It has to be raised about 2 inches off the bottom plate. Even with two men lifting and one screwing the holes in, we couldn't manage it. We tried a cantilever made of various tools, but nothing could hold the 300lb router with enough consistency to get the screws in. Fortunately, this was the loaner from the North Carolina ITEC. Even more forunate; they had the forethought to keep and ship the rack-mountable tray that the T640 can sit in. Once we saw that, we found our solution. So, the T640 sits 1RU off the bottom kick-plate. Unfortunate for us, we don't have our original trays in the deployed T640s, so we'll probably need to get an order in with Juniper right away. I need to check whether or not the Juniper loaner router that's at Nysernet has the tray.

We got the two Infineras racked up. Then re-racked when we realized we installed them backwards. Forgot that we're maintaining a hot and cold aisle. Fortunately, we hadn't done anything with the chassis yet.

The two fiber jumpers to the metro dark fiber weren't dangling in the Infinera rack, as we were expecting. Both of those jumpers get us to the I2 metro fiber between 600 W. Chicago and 710 NLSD. Eventually those will be used for the Infinera metro ring, but in the short term they'll be used to interconnect to MREN for the I2MM connectivity and to the legacy Abilene router in Qwest space. So, it's critically important that we get those up before we leave. If we don't, we won't have a way to get into the box to manage it. (this assumes that the POTS line cross-connect isn't there, which may not actually be a good assumption)

Hopefully we can finish up the physical install work tomorrow. I think it can be done. The config will take a bit more time, since I'm generating them on the fly.

BTW, Jay found something REALLY cool at Graybar that we're just now using. On the scale of useful POP tools, this goes right up there with the little yellow carts we use to move equipment around. He found a small plastic cylinder that dispenses very thin strips of labels with the numbers 0 through 8 on them. We're using those at both ends of a particular power cable to distinguish between them when they get laced together. In the old days, we used a magic marker. Ah, the future. It's so bright it might keep me awake tonight.....Nah. Off to bed for an early start at 8AM. Hopefully they have the badge PIN access worked out.

BTW, I left the camera in our car, so I don't have pictures. I PROMISE to get those out tomorrow since I know you're all on the edge of your seat.

EDIT: I've created a gallery of images from the install: Chicago Install Photos

Monday, November 06, 2006

The ugly face of circuit tracking


Long post on a long topic. The image above was taken from a brainstorming session this morning on how we should name circuits in the new network. At issue are all the different pieces and how to keep them together. There are three main things we need to keep track of:


  • The Level3 Infinera circuit IDs and the trunk fiber that "holds" the circuit
  • The I2 metro Infinera IDS (Chicago, New York and Seattle) and the trunk fiber that "holds" the circuit.
  • The fiber patches in the middle and at the end


So, you can end up with an end-to-end connection that spans dark fibers, different DWDM systems and rack-to-rack fiber patches. How do we glue all those together and how do we name them?

The Level3 circuit naming convention was going to be "I2-A-CITY-B-CITY-CUSTOMER-INDEX". So, a circuit from LA to New York for customer "Wackynet" would look like "I2-LOSA-NEWY-WACKYNET-00125". But those circuit IDs are unique to the DWDM system they run on. What we want to be able to give our customers is just a single circuit ID that our NOC can reference to find the more complete path with all the multiple "internal" circuit IDs.

Fortunately, that piece is almost ready to go. Ed Balas has been working on this for months and has a pretty darn good system in place. The new circuit database can be built in heirarchical layers. So, we can come up with a top-level circuit ID that can be drilled down into the sub-pieces. It also tracks the intra-pop fiber jumpers that each get unique identifiers. All of this is in place today, though not 100% production. Clicking on a circuit shows a nifty graphic with all the sub-pieces, color-coded by what type of sub-piece they are. It's all very 2007, in a 2006 world. My hats off to Mr. Balas for a very badly needed piece of software.

Now, how to name all those circuits. When we decided to put the customer name into the Level3 circuits, we locked ourselves in to changing the circuit ID if a particular A-Z connection is abandoned and re-used by someone else. You'd have to change the circuit ID just to get the customer name right. We started thinking more about that and we expect circuits to be abandoned and reused by other projects with more frequency than I had initially thought. So, the current plan is to remove the customer name from the Level3 circuit ID (with the exception of ESNET, since they have a more direct relationship with Level3) and provide the customer with a more top-level circuit ID that has a bit more metadata about the circuit's use. That top-level circuit ID will contain not only the Level3 Infinera system, but also the I2 metro piece (if part of the path) and all the miscellaneous fiber patches to get between the customer demarcs.

Pretty nifty stuff, huh?

It gets better. Not only can this system handle circuits in a heirarchical fashion, it can also handle sub-channels within those circuits. So, we can easily click on a particular 10GigE wave and see what VLANs are provisioned. Or we can see MPLS tunnels on an IP circuit. STS channels on a SONET circuit! Tracking them for documentation purposes is only one application. We can build maps and create monitoring tools that sanity check themselves against actual configurations. This is where the direct TL-1 access to the Level3 Infinera pool is really important.

Again, my hats off to Mr. Balas and our system engineering team for their work. It will really start to pay off here. I was freaking out a bit over the weekend on how much of this stuff is done in spreadsheets on my laptop harddrive. Now everyone else can play!

Prep for Chicago

We're knee-deep in the final preparations for the Chicago install. It sounds like there's some trouble with the ethernet patchcords possibly not being delivered in time, so we'd better be prepared to do a LOT of crimping. :-( We should be arriving in Chicago around noon. Gonna do a quick visit with Jake Sallee at 710 North Lakeshore Drive before we head over to the Level3 gateway to pick up our 600W Chicago building badges. Hope everything runs like clockwork...though it rarely does.

Friday, November 03, 2006

Chicago OOB tie-down

Tom Johnson was able to get the tie-down info for the POTS line in Chicago. Level3 should now be able to process their cross-connect to Chicago, so we may have OOB when we leave the install next week. Another engineer is working on the dialer configuration for us. Since we normally use ISDN line, we want to make sure we can route over these lines on-demand.

Thursday, November 02, 2006

Inside Delivery

Disturbing news from Infinera yesterday. Doesn't sound like they specified inside delivery on their shipment to 32 AofA in New York. That's a common issue, but one we try and avoid. Since Nysernet doesn't own the building, there won't be anyone to carry the box upstairs. We're trying to determine if Fedex will take it up or if they're going to leave it on the dock. Fingers crossed...

EDIT: Just got a note from Infinera. Sounds like Fedex will take the package to the 24th floor. A big thanks to Jay Underwood at Infinera for hunting this down.

MREN and Abilene interconnects in Chicago

Boy, how to describe this. So, in Chicago, we won't have our colocation ready at 710 North Lakeshore Drive. That means, we won't have the I2-owned Infinera DWDM metro ring turned up yet. But we will have the fiber pairs on the ring. We need to support two critical things for the members meeting:


  • 10GigE interconnect between the new Chicago router and the existing Abilene Chicago router
  • 10GigE interconnect to the MREN switch for the I2MM connectivity.


So, two pair of fiber can support two connections, assuming the optics can shoot the metro distance. The MREN one is easy since the MREN switch is located in 710NLSD. The Abilene one is a bit tricky. I2 owns a pair of fiber between 710NLSD and Abilene (at Qwest) for the HOPI-Abilene interconnect. So, we could glass that through 710NLSD to create a path all the way from Abilene to the new router. Unfortunately, the HOPI-Abilene interconnect is being used during Supercomputing. So, as a temporary measure, we're connecting the new router to the HOPI Force10 switch. That switch terminates the HOPI-Abilene interconnect, so we can bring up a VLAN across it. After the MM is over, we'll either leave the F10 in place, or we'll take down the HOPI-Abilene interconnect in favor of a fiber patch through 710NLSD. Eventually, in mid December, we'll get that 710NLSD colo ready and we'll turn up the Infinera ring.

Then, all these problems....go away.

The point of this posting is that I've got those interconnects at 710NLSD underway. The temporary Force10 interconnect was done yesterday and the MREN one should be done shortly. My thanks to Jake Sallee and Chris Young for their quick work here.

Check your optics

Always keep an eye on your optics. The 9 10G interfaces and SR optics aren't in yet for the observatory interconnect to the HP switch. I wasn't considering that a problem since I was just going to deal with that later with a remote hands ticket. But, there's talk about needing 4x GigEs for a members meeting demo (plans still being worked out).

So, we'll borrow from the 10G LR XENPAK pool we already have in stock. Since the Cienas won't be part of the MM, we can temporarily use it's LR optics on the HP switch. When the SR optics arrive, we'll need to swap that out. Or, if the Ciena interconnect is deemed more important than the Observatory 10G interconnect, we can swap it away earlier. Of course, this all has repercussions on the fiber jumper between racks. In the LR scenario, SMF fiber needs to be used. When we swap in the SR optics, we'll need a MMF jumper.

Wednesday, November 01, 2006

Chicago and New York Infineras shipped

Just got a note from Infinera. The chassis for the New York and Chicago metro rings have shipped. We're one step closer to getting our hands dirty with installs.