Internet2 Network Upgrade

It's been fun....

2007-10-09T19:37:00.000Z

As some of you already know, the Internet2 transition to Level3 has been complete for the past few weeks. The first clue was probably all the remaining Abilene nodes disappearing from the weathermap last week. They've been shut down and the first is coming out tomorrow. The other nodes will be removed over the next few weeks.

With everything complete, these little updates will cease. This was a wonderful experiment into the benefits (and perils) of direct communication between the Internet2 NOC engineering staff and the community. We learned a bit and may do more in the future in a more ongoing operational capacity, but don't look for that to happen anytime soon.

There are really too many people behind the scenes to thank. I'd like to acknowledge several individuals in particular that spent many sleepless nights on the road helping to make this transition a success. These are the true heroes of the upgrade that really made this all come together.

Hans Addleman
Andrew Lee
Caroline Carver
Jay Duncan
Tom Johnson
Aaron Pagel
AJ Ragusa

In addition the following people found other ways to assist. My thanks go out to them as well:

Steve Peck and Nathan Lucas for keeping the NOC service desk in the loop and assisting with procedural work
Matt Davy for pinch hitting for me when I was out in late March and keeping the CPS network on track throughout the upgrade
Caren Litvanyi for wading through countless spreadsheets and analyzing the minutiae of the suite buildouts
Luke Fowler and Ed Balas for their support on the database and documentation systems
Chris Small for keeping the observatory in step
John Graham for managing the entire Ciena network and alleviating hundreds of hours of concern
Jon-Paul Herron for keeping everyone's sanity at bay

This truly was a team effort and I'm proud to work with such high caliber individuals. If you see them in the hall at a conference, give them a pat on the back. They all deserve it.

So long....

When one problem exposes another

2007-09-14T05:41:00.001Z

Sometimes the network Gods have a funny way of toying with your mind. We noticed some L3 incompletes on the new CalREN circuit yesterday, and we pointed the finger at the optical levels on the CalREN interface. I worked with Level3 early this morning to clean the fiber path between I2 and CalREN. They found a bad port in the path at 600 Wilshire that was causing us to launch -14.3dB toward CalREN- just a hair under the tolerance specs of their card. After we cleaned that up, the L3 incompletes persisted. Turns out it was BPDUs that hadn't been filtered all along. Had they been filtered, we may not have caught the light level problem.

Someone was trying to tell us something. :-)

Anyway, CalREN is clean as a whistle now and should be shifting traffic tonight. I'm going to bed.

CalREN almost migrated in Los Angeles

2007-09-13T13:05:00.000Z

CalREN engineers shifted their traffic over to the new Internet2 link, and noticed a very small trickle of errors on the port. Light levels are showing up right below the threshold, so we'll need to do a bit of cleaning there. Look for that to happen soon.

NGIX-W/NREN migrated to Salt Lake City

2007-09-13T03:20:00.000Z

I migrated the Sunnyvale NGIX-W and NASA connection over to the Salt Lake City router tonight. Always a bit tense when you move that many peers, but this went as smooth as silk. A big thanks to our friends at ESNET who provided a concurrent GigE interface to the Ciena in the Sunnyvale POP so that we could backhaul the circuit to Salt Lake City.

HOPI IP migrated in Los Angeles

2007-09-11T01:26:00.000Z

We were able to get the HOPI 10GigE interface up in Los Angeles today, across our metro fiber between 600 Wilshire and 818 W. 7th. We tried to get the PAIX 10GigE up to a loop in Sunnyvale, but we're having some metro problems in LA. The signal just ain't getting between the buildings. We have a ticket open to troubleshoot that in the AM.

LEARN migrated UT Austin behind them on Friday night, so they're now shut down on Houston. That leaves three BGP sessions left on Houston.

I need to schedule CENIC on LA- the last BGP session on the old Los Angeles router.

After that, it's just PAIX, NGIX-W and Oregon in Sunnyvale. I'll probably move the NGIX/Ames VLANs this Wednesday, along with PAIX if I can get them up tomorrow. Oregon may be a few more days out than that.

Pacific Wave, CENIC and LEARN and all the rest...

2007-09-06T03:21:00.000Z

It's been a month or so. Let me see if I can catch everyone up. We've been mainly busying ourselves with getting some fiber patches and circuits lit on the west coast.

I just migrated all of the Pacific Wave Los Angeles peerings over to the new Los Angeles router. Once we got the link up, it was a painless hot-cut.

We did some metro fiber troubleshooting in Los Angeles this evening and the CENIC/HPR connection is up to the new router. We'll need to coordinate a time to swing their peering over to the new circuit.

LEARN a few weeks ago, and the Texas schools are all working on aggregating behind them. Our piece is done.

USF also looks to be up in Atlanta, so we'll probably be able to shut the old M5 down pretty soon.

We're close on Oregon and PAIX. I'm hoping to get all our pieces worked out this week. There are some interfaces being shuffled around in LA to get them from the old router to the new one. Those should be in place by Friday. We very well could have everyone off the old routers by the middle of next week, if things click. Probably more realistic to expect two more weeks, though.

Houston T640/Ciena installed

2007-07-26T12:53:00.000Z

The Houston T640 was installed in Houston yesterday and the backbone circuits were lit through the site. Our engineers still have some other work to do today to get the Ciena and PCs installed. Overall, things appear to have gone fairly smoothly.

Denver Abilene router shut down

2007-07-12T02:57:00.000Z

Denver was one of my favorite routers. Mainly because I love Boulder so much and always got to spend some time out that way. I feel a bit like the delivery doctor and the executioner on this one since I helped install it just under four years ago with Caroline. Debbie Montano stole a few moments away from the Qwest corporate offices to see how things were going. Unfortunately, we gave her a camera and posed for a few photos.

The sad thing is that I have too many pictures of me standing next to network equipment with that same cheesy grin on my face, like I just caught a big bass and am holding it up for the camera. At least Caroline looks like she's putting away some tools or something.

Anyway, Denver hasn't been doing much for anyone lately. All networks were moved off by last week and the backbone links weren't in the path for much of anything. It's been turned down, so our engineers can (gingerly) move it to Houston. We need the Denver power controllers in Houston, so we're going to allow Level3 next week to get them installed for us.

Ah, Past Chris. If Future Chris could tell you anything, I'd probably make sure you know how to lace better. And that you should never look away from your hands to pose for a photograph when you're that close to live DC power. Sigh

Los Angeles T640 installed

2007-07-05T01:09:00.000Z

Tom and Hans were able to get the T640 installed in Los Angeles fairly quickly. The link to Kansas City is up to provide IP connectivity while we wait for the high speed IP drop from Level3. The OC-192 to Seattle didn't come up, but that's likely because we have a fiber backwards in the path. We didn't have light to the Westin building when they were running the fibers in the fiber meet me room, so there was no way to judge the polarity of the cable.

I'm taking the next few days off, but Andrew is going to work on getting that circuit up while I'm out.

Denver is coming out next week. Turning it off on Wednesday night....

Los Angeles! Albuquerque! Denver! The updates! The updates!

2007-07-02T14:33:00.000Z

I've been getting many a query from you Internet2 upgrade blog junkies asking whether I was dead or had just turned my back on this whole ballyhoo. Fortunately, neither. I'm very much alive and things have very much been moving forward. I had anticipated a bit of a lull over the last few weeks, but I was (sadly) mistaken. Just a rundown of the big things:

Los Angeles
Savvy I2 blog readers will recall that the Los Angeles router was slated to be installed two weeks ago. We pushed that out a week to accomodate for some power controller installations. We really hate doing DC power from the house since we don't control both ends. So, we went out last week. But there was one, glaring problem: no router. There were a whole host of scheduling mishaps between Fedex and Qwest in Kansas City that meant our router didn't get shipped the prior week. We finally got Fedex on-site on Wednesday to ship it overnight to Los Angeles, but fate decided to stick her long ugly nose into things. There were actually two crates at the Kansas City POP. One contained the router that was going to Los Angeles using Fedex Express Freight. (Note the italicizing of Express Freight. Very important.) The other contained the reclaimed flotsam from the Abilene rack that was going Fedex Freight back to Bloomington. Unfortunately, both got picked up by regular Freight, which meant no guaranteed delivery times in Los Angeles. Up until the very end, it looked like things were going to work out, but it wasn't meant to be. Fate 1, Upgrade Team 0. So, we're sending two engineers back out tomorrow to finish the install and commission the node. They're going to miss all the area fireworks, so they deserve a round of applause

Albuquerque
Though the Ciena was installed in Albuquerque in early June, the phone line proved to be more troublesome than we thought. The Qwest phone group is totally separate than the Qwest transport group, meaning that it was essentially two different companies talking to each other. That slowed things down to the typical speed we'd see with most phone line installs. Add the fact that the Albuquerque POP is essentially a bundle of disjoint buildings in the desert, and you have a recipe for slowness. Fortunately, Tom Johnson stayed on top of things and we got the phone line delivered last Tuesday. The cross-connect into our equipment was completed on Friday. In the interim, ESNET graciously helped us out with a Layer2 VPN to the Washington DC POP so we could commission the node. (Recall that we don't turn the Ciena nodes up. Ciena does.) That paved the way to get the University of New Mexico connected and working up to Salt Lake City.

Denver
Aside from UNM, there were a few others on the Denver router, which we plan to pull out next week. Both Front Range Gigapop and the Utah Education Network are up on the new router and passing traffic. There was a bit of scare when the fiber cross-connect in SLC wouldn't allow us to ping the remote ends. Turns out it was a hardware problem with our router card. Got it replaced and everything looks spiffy

***

So what's next for our intrepid crew? Not too much left, really- in terms of physical work. We're yanking Denver next week. It's being sent to Houston where it will be installed two weeks later (to allow time for the power controllers to be installed). After that, there's a pending Ciena install in Seattle, but we don't have a time frame nailed down for that. There's obviously a lot of work left to transition customers to the new network in LA and Houston, but most of that is something that can be done remotely. Of course, we have to pull out the Sunnyvale, LA and Houston routers at some point in the next few months. I'm working on solidifying that schedule right now.

Kansas City router begins journey to LA today

2007-06-14T12:35:00.000Z

Our faithful T640 in Kanasas City (Qwest) will begin its magical journey to Los Angeles today as Jay and Andrew yank it out. I turned the chassis down last night. For some reason, the data collectors that are polling the Abilene routers have tanked, so that explains the state of the weathermap. I'm assured that it will be fixed soon to reflect the current topology.

Level3 by the numbers

2007-06-11T15:03:00.001Z

With the Level3 build of the Internet2 network largely completed, they shared some fun stats with us:

13,500 Long Haul route miles
Deployed and configured over 300 Infinera Network Elements
Day 1 capacity of 100Gbs
Built 27 custom Colocation Suites representing 3,365 SQFT of space including:

94 Racks

60 Individual bulk cables with 48 & 96 fiber count

Deployed 64 metro fiber route miles
Enabled connectivity from Houston-to-El Paso & El Paso-to-Denver for L(3) to Internet2/GRNOC and ESnet NOC access
Developed the Virtual Network Operations Center – Provisioning and Troubleshooting Dashboard
Product Summary: Wavelengths, HSIP, Private Lines, Metro Fiber, Colocation, Cross-Connects, Power Controllers, and GPS timing

A job well done by Level3!

But now's not the time to rest on our laurels. We still have work to do on the IP network migration and Ciena network turnup. Kansas City turndown this week, and Los Angeles install next week. Does the fun ever stop? :-)

How sweet it is!

2007-06-07T23:14:00.000Z

This is the week of good news.

A bit of recap and a few other bits of news:

SLC to Seattle OC-192 up across metro fiber
Level3 finished the DWS to Los Angeles
Albuquerque and Sunnyvale install done, meaning all Ciena-only sites have been deployed
the University of Memphis circuit is up and ready for route exchange
Kansas City can come out next week
LA suite delivered a bit ahead of schedule and will be accepted on Tuesday

The crew in Albuquerque deserve an extra round of applause for working on 3 hours of sleep and handling things so well in one of the most punishing of sites.

early travel logistics problems in Albuquerque

2007-06-07T15:14:00.000Z

Our engineers in Albuquerque today decided to jump start the fun by scheduling themselves onto a flight that got cancelled. Jay just sent an update that I'll let speak for itself:

"Landed in Denver, which was bumpy but fine. Found that the flight to Albuquerque cancelled due to wind and the next available flight to Albu, 2:46pm Thursday. Time to Albu driving, according to airline [gentleman], was 4.5 hours. So we rent a car. And begin waiting for our bags to be pulled off the plane. 3.5 hours later, we're told that no bags were forthcoming. Awesome. We hit the road, and find that it's more like 6.5 hours. Bad research. Arrive at hotel 4am local.

Okay, so we're here, but our bags aren't, which means we have no boots, and no hard hats. "

Internet2: From Sea to Shining Sea

2007-06-06T19:20:00.000Z

Yeah, so Internet2 made my 3 month old daughter cry this afternoon.

More to the point, I made my 3 month old daughter cry when I leapt for joy at seeing the Salt Lake City to Seattle OC-192 come up. I still blame the OC-192.

We've been fighting with this one for the past week or so, getting jumpers run and the appropriate patches in the fiber meet me room at the Westin Building. In the end, all worked out well and Internet2 has SONET connectivity from the burroughs of New York to the shipping yards of Seattle.

I hope to get things configured very shortly to begin passing traffic through the new router in Salt Lake City. Now that it's dual homed, we can start moving folks over.

With that, the last two Ciena-only nodes, and the University of Memphis OC-48 in play this week, it could turn out to be a very nice week for the project.

Denver and Baton Rouge Cienas installed

2007-06-05T12:12:00.000Z

Our engineers installed the Baton Rouge and Denver Cienas late last week. Had a bit of trouble in Denver with our out of band router being dead on arrival, so I'll get a replacement out this week. Otherwise, things went well.

This week is Albuquerque and Sunnyvale.

Did you miss it? Salt Lake City installed last week

2007-05-30T22:19:00.001Z

Apparently our network elves have been hard at work in the wee hours of the morning cobbling shoes and installing router nodes. Unbeknownst to many, the Internet2 network peeked its head into the Beehive State and decided to set up shop. It's safely tethered to the Kansas City router now, but we're aiming to change that in the next week with a second lifeline to Seattle. That will make it a full-fledged grown-up router, able to route with the best of them.

Steve Corbato dropped by on Friday just to make sure our boys hadn't electrocuted themselves and posted some photos of the visit. I know what you're going to ask, and the answer is: "No, the POP doesn't have a brew-pub in one of the suites."

For some reason the router still thinks its in Atlanta. :-) I''ll be configuring it tomorrow so it can play in all the reindeer games with the rest of the big boys.

We have two teams heading out West and South to install Denver and Baton Rouge Ciena nodes. We wish them well.

Kansas City backbone and GPN transitioned

2007-05-18T20:47:00.001Z

I'm happy to announce that the Kansas City-Chicago and Kansas City-Atlanta OC-192s are fully in production on the new network and are routing traffic between the three cities. In addition, we now have a new interconnect between the old Abilene network and the new Internet2 network in Kansas City.

Concurrent with that, GPN migrated their 10GigE over to the Level3 POP.

We expect to have the Salt Lake City T640 up next week with connectivity back to Kansas City and (possibly) to Seattle.

Have a good weekend.

KANS T640, GPN transition, El Paso, Atlanta

2007-05-17T22:09:00.000Z

Rest assured, we've been diligently working this week: on not-so-public/exciting/sexy install packets, shipping and documentation. It's all been prep for today, when four major tasks hit.

The Kansas City T640 is fully configured and has backbone links to Atlanta and Chicago. It's been brought into the iBGP mesh and I'm putting the finishing touches on the monitoring/documentation. The Chicago circuit came up yesterday, and the Houston circuit (actually two circuits glued together in Houston in anticipation of the eventual Houston T640 install) was turned up 10 minutes ago.

Second: we nailed down the logistics of the GPN transition to the new network tomorrow. GPN will be moving their router from Qwest to Level3, thus abandoning some dark fiber between the two POPs. Being the opportunistic people that we are, we're both using that fiber to maintain some connectivity into the Qwest POP using the same CWDM splitters we used in Atlanta. The clock starts ticking at 9AM tomorrow on the move. Our side is fairly straightforward, and I'm sending good thoughts toward the local engineers that have to lug the 7600 between POPs.

Third: The El Paso Ciena install is underway. All equipment is on site and I heard good things from the field on the progress made. That should wrap up tonight or early tomorrow morning.

Fourth: The Atlanta T640 was pulled out of the Qwest space and will shortly begin its wonderful journey across the country to Salt Lake City for install next week.

I love it when a plan comes together.

Kansas City Install almost complete

2007-05-11T12:44:00.000Z

With the exception of a few power runs and a bit of documentation, the Kansas City install is largely complete at the Level3 POP. The T640 arrived yesterday and was powered up. We'll be activating the backbone circuit to Chicago today. The circuit to Atlanta (temporarily bypassing Houston until it gets installed) will have to wait for Monday when we slot the OC-192 I had yanked from New York on Thursday).

We're solidifying the details, but there will be an interconnect between the old Abilene router and the new Internet2 Network router via some University of Nebraska metro fiber. We'll be putting another one of those CWDM optical combiner/separators on the link to share the path with GPN. Not sure on the timing of that yet, but it's starting to look like we'll make that happen next Friday. Once that's complete, we'll drop the Qwest OC-192 between the Abilene Kansas City router and the Internet2 router in Chicago. That frees up some much-needed OC-192 PICs and allows us to move forward in Salt Lake City in two weeks' time.

DC router removal saga

2007-05-09T00:55:00.000Z

You may be wondering why I didn't gush over the perfect router removal we had on Monday. That's because it wasn't meant to be. Our engineers' flight to DC got cancelled on Monday morning and they couldn't get rebooked until later in the afternoon. So, they rented a car, put on some tunes and did a Tommy-Boy road trip out to DC. Knowing Jay, there was probably a lot of Rush involved.

Jay and AJ got to the Qwest POP this morning only to find that the empty T640 crate had been "obliterated" in shipping. This is where the driving came in handy. They had a smaller T320 crate that they were able to use for the T640, and the put the rest of the de'commed equipment in their Taurus. Sadly, no more room for Rush.

Unfortunately, the work at Qwest took them pass their badging window at Level3, so they need to stick around another day so they can complete some PC install work early tomorrow morning. Then it's back home to Indiana.

We're attempting to secure another T640 shipping crate from Juniper, but if anyone has one that they've been using as a doghouse or Tornado shelter, drop me a line. I'm off to check Craigslist.

Atlanta and Washington DC Abilene T640s turned down

2007-05-07T21:42:00.000Z

I turned down both the Atlanta and Washington DC T640s just before 1PM EDT this afternoon. This officially removes them from the network, though they stopped routing traffic last week when we moved the Atlanta-Houston circuit over to the new router in Atlanta.

Washington DC is being pulled out tomorrow. It was originally slated for today, but our engineers' flight to DC was cancelled and they won't be able to get in before 5PM this afternoon. That router will make a quick trip to Kansas City for install on Wednesday.

Jacksonville Ciena installed

2007-05-04T19:32:00.000Z

Just a quick note that I got a call from Tom Johnson to tell me that they had finished the Ciena install in Jacksonville.

Now I'll stop posting on Atlanta

2007-05-04T02:50:00.000Z

And now, drumroll, please....

Atlanta.
Is.
Migrated.

Yeah, I can hardly believe it. The past few days have been up and down (mostly down). Yesterday, I'm ashamed to say, I forgot that the GigE port on the M5 that we were swinging the SOX dark fiber to was LC. Of course, the SOX card has SC connectors and the Qwest techs couldn't locate any SC-SC barrel connectors. So, we had to move back the migration to this evening, while we overnighted an SC-LC jumper and connectors to Atlanta.

Trouble is, we got our wires cross and accidentally sent it to the Level3 POP in Atlanta. We were able to locate a local courier that could drive it between the two POPS, so that turned out to be a non-issue, but one that had me on pins and needles for a few hours.

The next problem was port the fiber was plugged into in the Level3 POP. When Jeff was in Atlanta two weeks ago, I had forgotten that we slotted the LH SFP into port 1 of the 4-port GigE card instead of port 0, like I had planned and wrote in the interface description. When I devised the instructions to Level3 for running the fiber on that side on Monday, I put the wrong port down and completely missed it. Fortunately, while away from the office today at 3, the thought magically popped into my head and I had a coworker double check my work. Turns out, I was indeed mistaken, and we had a short window to get Level3 out to the POP. It was 4:30 and we were still waiting on both techs to call in. Fortunately they did...at the same time.

Magically, the connection came up. I say magically, because at this point, I'd probably believe you if you were to tell me there was imps running around in our fibers just trying to keep Atlanta as-is.

All that was left was to migrate USF down to the M5 at 10PM and do the backbone move at 11PM. The USF move went well, though there was some sort of problem with the adjacent PIC to the University of Mississippi. For some reason, all the BGP sessions through that ATM card went down and IP reachability was nil. None of this work should have affected them, but I reset the card anyway. That fixed it for some reason. Hmmm. Need to check that out in the AM.

The backbone move went as smooth as butter. Down on one router, back up on the other within 2 minutes. Even the Qwest techs were amazed at how easy it was.

So, this feels a bit like an Oscar speech, but we really have to extend our thanks to a lot of people. Scott Friedrich at SOX, and all their work really made the difference. Qwest did an awesome job of testing the circuit in advance and working around the multiple changes in our schedule. Level3 was spot-on with the on-demand remote hands and help today with the courier. This was a real team effort and one of those war stories that will linger in my mind forever.

Onward to Kansas City!

Last night's fiber cut

2007-05-02T14:16:00.000Z

Now that the dust has settled on last night's bridge fire, we'd like to extend our gratitude to Level3 for making such a rapid repair. We've dealt with bridge/tunnel/road/railway fires before, and this was probably one of the quickest repair times we've seen for such a disastrous event. This also highlights the flexibility we have to route traffic around such events. Kudos to all involved.

Boston circuits update

2007-05-02T08:01:00.000Z

Level3 rolled everyone to a new fiber path in Boston shortly after the last update. All circuits on that path are back up and working. I believe the response time was just under 4 hours. We know they migrated more than just the Internet2 Infinera network over to alternate fiber paths. Pretty impressive footwork on pretty short order to get around a severe problem.

Thank You for Not Smoking.

Update on the Boston bridge fire

2007-05-02T03:01:00.000Z

While working through the motions of getting the Boston waves routed around the other side of the ring via DC, Level3 informed us that the outlook on the downtime isn't as bad as they initially expected. They're working toward routing their network across some diverse fiber, as well as trying to fix the fiber under the bridge. The prognosis is now on the order of hours, and not days. We're expecting to have partial or complete restoration in the next several hours.

SOX migrated in Atlanta

2007-05-02T02:56:00.000Z

All of the networks behind SOX have been successfully migrated in Atlanta. Tomorrow we'll be swinging the Houston OC-192 from the Atlanta Qwest router to the Atlanta Level3 router.

Question: When can a cigarette take down your network?

2007-05-02T01:48:00.000Z

Answer: When you throw it at a bridge and light it on fire.

Looks like all our circuits between Boston and New York are down due to a fire under the Longfellow Bridge in Boston. Read all the fun here: http://wbztv.com/topstories/local_story_121202334.html

Artists rendering of the moment our circuits went down

Level3 is telling us 1-2 days for service restoration right now, though they information could change as time progresses. We'll look into whether or not Level3 can re-route the Chicago to New York OC-192 through DC until the Boston path comes back.

There were a few updates after this:

Update on the Boston bridge fire

Boston circuits update

Last Night's Fiber Cut

Looking better in Atlanta

2007-05-01T18:29:00.000Z

We're taking another stab at Atlanta this week. We were hoping for a third pair of metro fiber to be delivered in time for the new attempt. When that didn't look promising, we tested out a new pair of optical splitters last Friday and overnighted them to Level3 and SOX. I just got off the phone with both Scott Friedrich (Georgia Tech) and Level3. The new splitters have been inserted and we brought up link to a second 10GigE LR port on the SOX Force10. Here's the next order of events:

Tonight, I'll be migrating SOX over to the new router. They've already pre-configured the VLANs on the new port, so it's just a matter of deactivating one config and turning down another.
Tomorrow, during the day, I'll work with Qwest to move the abandoned SOX fiber over to the M5 to bring up the GigE LH link to the new router.
Tomorrow, during the night, I'll be switching the M5 over to the new fiber, migrating the USF OC-3 down to the M5 and moving the Qwest HSTN-ATLA backbone circuit to the new router.

I'm starting to get superstitious about these sorts of things, so if I could ask everyone in the R/E community to cross their fingers for us, I think we'll be OK. If things don't work, I'm going to blame YOU! :-)

The long and short on Atlanta

2007-04-27T15:37:00.000Z

Well, as most of you have surmised from the I2 notifications that went out, the work in Atlanta did not go as planned last night. The short answer is that it came down to a bad optical splitter, thus halting us at step 1 of the 8-step transition I had planned out. Because of this, Washington DC needs to stick around for at least another week. For those that are wanting to hear the saga, read on...

We got confirmation that all the cross-connects had been run in the 56 Marietta FMMR on Tuesday. I had Level3 slot cards for me on Wednesday to do some pre-testing. We immediately ran into problems. There was some confusion on the panel assignments in the FMMR that were resolved early Thursday morning. So, we spent all day Thursday working with SOX and Qwest on getting the fibers cleaned up to the point where we could continue with the transition later in the evening.

First, on to Qwest. They've been absolutely fantastic in keeping in touch with us on getting things pre-tested. Once we were ready on our end, we plugged the OC-192 into our router and were testing to a loop. A bunch of us were under the impression that there was active electronics at 55Marietta. When the network was initially installed in 2003, there was a Ciena box at 55M that was extended over the long metro jump to Forest Park. Somewhere along the line that changed, and the circuit no longer passed through 55M. We didn't discover this until I got on the phone with the tech and he said the path was glass all the way through 55M to Forest Park, some 12 miles away. It was no wonder I couldn't see the hard loop they placed for me. I couldn't shoot that distance there and back. The put a test set on it and we did a head-to-head test. I could see them and they could see me, though we're right on the receive sensitivity margin. The circuit was 100% clean, though, so all looked well. First crisis averted.

The SOX connection was a day-long struggle to remove attenuation from the patches between Level3 and Georgia Tech. To test things out, we temporarily plugged a 10GigE LR interface directly into the metro fiber on both sides, bypassing the optical splitter. The initial reading on the Georgia Tech side was -13dB- enough to get the link up, but probably not enough to shoot through the attenuation that would come with the insertion of the splitters. By days end, we had the signal strength up to -9dB. Second crisis averted.

The Internet2 engineer that was going to be stationed at Level3 during the change window phoned me up at 4PM and said he couldn't find the optical splitter in our suite. While he was locating a tech, I was mapping out directions to drive down to Atlanta to get it to him early in the morning. Fortunately, they located it just before 5PM and all was well. Third crisis averted.

So, I went home and spent the next three hours pre-configuring the routers with all the interface, BGP, MSDP, etc configuration that needed to be done. 10PM rolls around and everyone dials into the bridge. I don't know how many of you have ever had a bunch of people in colo facilities on a phone bridge at once, but it's almost impossible to talk. Jeff and I had to connect on a separate phone line since we couldn't hear each other.

We started out with the first step: moving SOX over to the optical splitter. No link. -36dB received on the I2 side. We did a bunch of different testing, including removing the splitter from the path, swapping out all the fibers, and jiggling connectors. In the end, the splitter was adding 9-10dB of attenuation in both directions. Well, it isn't supposed to be doing that, but of course, we had no other way of proceeding. So, we backed out of everything. Cancelled the SOX, M5 and USF move with Qwest. Cancelled the backbone circuit move. Cancelled the Washington DC router pullout (and called the engineers who were going to get up early in the Friday AM to fly out).

We have some alternate plans. There's actually a third pair of fiber we ordered a while ago that should be ready for our use if we can get the tie-down info in time to order a cross-connect. We can also shim HP switches in the middle and plumb the GigE through SOX. (SOX, unfortunately, doesn't have any long-haul 1G optics for their Force10, which was something we considered a while ago) I'm hopeful that we can get the fiber since that's the easiest way with the least moving parts, but if we need to use something else, we will.

Right now, I'm aiming for a transition next Wednesday night. Once that gets more solid, I'll let everyone know.

Washington DC almost definitely coming out on Friday

2007-04-23T22:50:00.000Z

Got another bit of good news. The FMMR cross-connect between Qwest and Georgia Tech has been completed, which means all the patching is in place for the Atlanta-Houston OC-192 groom on Thursday night. We've scheduled travel and I'm arranging to do some testing. Still need to verify the end to end path, but we're headed in the right direction.

University of Louisville 10GigE up on Atlanta

2007-04-23T22:48:00.000Z

Yeah, so I'm stuck in the Indianapolis airport waiting on my delayed flight to Washington for the I2MM when I got a note that made me very happy. Caren tells me that the University of Louisville's 10GigE connection from Louisville to Atlanta is up and passing traffic. This was one of those connections that ate up a lot of space on our e-mail server, so it's one that I'm happy to see completed.

Now, if my plane would just arrive the day would be complete.

Tulsa Ciena and DC removal

2007-04-19T17:36:00.000Z

The installs have resumed a bit this week with Tom and Hans slotting the chassis today. So far, so good, as I've not heard of any issues from the field. With luck, they'll be finished by day's end and ready to come back.

We're currently targeting next Friday (4/27) for the Washington DC removal. We did receive the final LOA/CFA we needed from Qwest yesterday in Atlanta, but there were too many unknowns in terms of scheduling and heroics needed to get crates shipped around that we decided to not to move the Atlanta backbone circuit this week. With the Members Meeting next week, we decided that the earliest we could schedule it would be next Thursday. Hopefully the move will go well and we'll be able to yank the DC chassis the next morning. That will put us on track to install Kansas City the following week.

Washington DC node removal status

2007-04-16T19:57:00.000Z

I've gotten a few questions from the community about how the Washington DC router removal is going. We're unfortunately still waiting on some documentation and an order to process for a FMMR cross-connect. While we continue to reach out to our partners, I don't have any ETA on when that will be ready. For now, the IP network transition is on hold until we can get that jumper run. Funny how a 2 meter piece of fiber can hold you up....

We're still looking into alternative transport methods and am getting more inventive as we go. Hope to have more shortly. At this point, I'm doubtful it will happen this week. Even if we can get the proper LOA/CFA, it will likely take several days to get the cross-connect made in the FMMR. It also takes a minimum of 72 hours notice to get remote hands lined up- none of which I can easily arrange until I have some level of confidence that the jumpers will be run.

But hey, I've been wrong before. Maybe things will start to click.

So why hasn't the Washington DC router moved yet?

2007-04-12T14:20:00.000Z

In a word: Atlanta.

Have a look at this image to the right, which depicts the current state of the network, with both the Qwest and Level3 backbone links. Note that the old Abilene network (with routers in blue) and the new Internet2 network (with routers in red) are interconnected in New York and Chicago. Those two links are keeping the two networks talking to each other.

If we were to remove the Washington DC router, we would also have to turn down the DC to New York Qwest OC-192, which would mean the two networks are only interconnected in Chicago. Unacceptable.

A few months back, we looked at two different options to keep the networks dual-homed. The first was to do something similar to what we did in Indianapolis: have Qwest patch the southern route into a single Houston to New York circuit. The second option was to figure out a way to create an interconnect in Atlanta. Because we didn't like the idea of traffic having to go all the way to New York to get between the two networks, we focused on interconnecting in Atlanta. Stay with me.

Atlanta has a few additional quirks that make the transition a bit more difficult. Ever since the 2002 migration to the Juniper platform, we've had a small Juniper M5 providing connectivity for the University of Mississippi OC-3 ATM (the T640 platform doesn't have small ATM interfaces). Mississippi isn't quite ready to move on our schedule yet, so we need to leave the M5 at the Qwest POP, with the Mississippi circuit still connected. So, we need to interconnect that M5 with the new T640 at Level3. There's a good 20 miles of space between the two POPs.

Fortunately, Georgia Tech is abandoning a pair of fiber between 56 Marietta and Forest Park (Qwest). Internet2 has two pair of metro fiber between 56 Marietta and 180 Peachtree (Level3). Georgia Tech's connection will migrate to the Internet2 metro fiber, thus freeing us up to use their old 20 mile run out to Qwest from 56M.

The original plan was to put the M5 GigE interconnect on one pair of fiber and SOX on the other pair. But then we realized that we could swing the Qwest Houston to Atlanta circuit up from the Qwest POP in Atlanta to the Level3 POP in Atlanta. That circuit passes near 56 Marietta, so we started down the path of figuring out how to cram three signals onto two pair of fiber. The solution was a wideband CWDM passive optical splitter. The SOX and M5 GigE will share a pair of fiber and the Qwest circuit will ride the other.

Even though I know that's crystal clear and no one could possibly be confused, I made up a somewhat irrelevant diagram to explain the obvious. :-)

Now that that is crystal clear, I'll explain the holdup. Raise your hand if you've ever had a fiber meet me room (FMMR) cross-connect work perfectly the first time around. I see five hands. When I discount those of you that are lying, I see one- and that person probably works in the FMMR. It's typical to encounter delays on getting panels right and making sure the person has the appropriate authorization to plug a fiber in. Each carrier wants it done a little differently. For example, Qwest wants the tech to dangle the fiber, but not plug it in. You need to open a ticket to have Qwest go out and plug the fiber in. None of this is particularly wrong- just difficult to navigate sometimes. That's what we're running into here.

We were hoping to do all the transition in Atlanta last night, but we've pushed it back to next week. That will delay the Washington DC router removal by a week. Since the DC router is being shipped to Kansas City, the KC install will likely also be pushed back. We have a bit of cushion on the schedule, so it's possible that things may accelerate past that, but probably unlikely.

So there you have it. I was clearly misinformed when I believed that Chicago was going to be the most complex transition in the network. That's looking like a campus wiring closet switch install compared to Atlanta. :-)

NGIX-East migrated to new Washington DC T640

2007-04-12T14:18:00.000Z

Caren moved all the NGIX-East peerings from the old Washington DC router at the Qwest POP to the new Washington DC router. All went well.

Along with the Drexel and PSC moves earlier in the week, that means everyone is off the old router and it can be turned down...

Well, almost. I'll explain in a bit...

Chicago to New York IP Backbone Link up !

2007-04-06T17:47:00.000Z

Yesterday afternoon I brought a new OC-192 IP backbone link over the Level3/Infinera system between Chicago and New York via Cleveland and Boston. After our standard testing, this was put into production around 1pm today. The fiber path for this circuit is actually longer than the fiber path from Chicago to Washington D.C. to New York. Since we use the fiber route miles as our IS-IS metrics to provide the lowest latency path, this will only be used as a backup path for now.

This connection is important to the transition because it allows us to remove the old Washington T640 without leaving the New York T640 single-homed.

Boston Ciena node installed

2007-04-06T14:29:00.000Z

The Boston Ciena node was successfully installed yesterday by engineers Andrew Lee and Hans Addleman. With the exception of a bloody mishap with the tools, everything went well. :-) Ciena technicians are now able to come in and turn the node up.

Indianapolis T640 removed from service

2007-04-06T14:22:00.000Z

Matt Davy was successful in turning down the two remaining backbone circuits that pass through the Indianapolis T640 this morning. The old KSCY-IPLS and IPLS-CHIN circuits are now fused together in Indianapolis to create a Kansas City to Chicago Qwest circuit. The Indianapolis T640 is now completely disconnected from the network and will ship back to Juniper this Monday. Another round of thanks to Qwest for their assistance this morning.

U of Louisville temporary connection to the Indiana Gigapop

2007-04-04T13:31:00.000Z

The time has finally come for the loaner T640 router to return back to its home. As many of you know (don't you!) that router's been passed around the network a few times, where its been needed. It's been providing connectivity in Indianapolis for the past few months, while we sent the old Indianapolis router to Washington DC.

Early this morning, Caren and Jay moved the last connector, the University of Louisville, off the Indy router. While their permanent connection will be a 10GigE to Atlanta, they're still waiting on some fiber work to be completed in Louisville. Fortunately, they agreed to temporarily move behind the Indiana Gigapop so as not to hold of the return of the Juniper router. With Monday's migration of the Indiana Gigapop to Atlanta, we freed up a pair of fiber between the Gigapop and Qwest. Louisville was regroomed to that pair. It's everyone's hope that this only lasts a short time. While, we're happy to host them, we know they're anxious to get moved over to their dedicated connection.

We should pause and give our extreme thanks to Juniper for their generous assistance over the past five months. We wouldn't have been able to move the networks in place without their support.

The Indiana Gigapop migrated to Atlanta

2007-04-04T13:28:00.000Z

This past Monday evening/Tuesday morning, Matt Davy successfully migrated the Indiana Gigapop to its new 10GigE interconnect to the new Atlanta Internet2 router. This frees up a pair of fiber from the Indiana Gigapop to Qwest that will be temporarily used by the University of Louisville.

Catching up...

2007-03-21T15:48:00.000Z

We apologize for the short intermission in our blogging....we hope to get regular posts going again. Here's a brief summary of what happened over the break....

March 1st: Level3 delivered the Infinera spans from New York to Boston to Cleveland on schedule.

March 2nd: Level3 delivered the Infinera spans from Washington to Atlanta to Chicago on schedule.

March 8th: All connections transitioned off of the T640 at Qwest. The Abilene node at Qwest was removed and the T640 sent to the Level3 POP in Atlanta

March 15th: The T640 shipped from Chicago was installed as the new Atlanta router and the Infinera backbone circuits from the new Atlanta T640 to Chicago and Washington D.C. were brought up.

March 16th: Level3 delivers our SONET timing source in Chicago which allowed us to turn up a complete Ciena SONET system from Chicago through Cleveland, Pittsburgh, Washington, Philadelphia and New York and then a ring from New York back to Cleveland via Boston. We should have a 100% fully functioning Ciena network on these paths by the end of this week !

Coming up....

We'll be do installs in Indianapolis, Louisville and Nashville this week and next. We plan to remove the T640 node from Indianapolis on March 30th and plan to remove the T640 from Qwest in Washington the week of April 2nd - it will be heading to a new life in Kansas City.

Nysernet migrated in Chicago

2007-02-13T03:44:00.000Z

I worked with Bill Owens this evening to move the Nysernet Chicago peering to the new router. All looks well. Just a few more networks on the old Abilene router to go before we pull it out next week.

Don't feed the engineers

2007-02-13T00:54:00.000Z

We received a welcome package of frozen ribs, wings and pie in the mail from Rob Vietzke and company from Internet2 last Thursday. Apparently they like the work we're doing. We decided to make an afternoon out of it and had a pitch-in. If you're wondering how this is related to the upgrade, just think about how stable the network was on Friday and what part this lovely surprise played in that. Bravo, Internet2! This was a fun lunch and we managed at least five minutes without talking about the upgrade. :-)

Some brewing discussions

2007-02-12T19:33:00.000Z

So, it's Joint Techs week, which means you'll see no MAJOR changes to the network. That doesn't mean we're just sitting around watching the snow pile up, though. (Even though I'm told, it's a fascinating sport these days, in New York). We're working on getting BGP up to a few networks this week. Most of them will probably come up without an exchange of routes, just to avoid any routing weirdness for the folks in Minneapolis:

Nysernet [Chicago]
Wiscren [Chicago]
Merit [Chicago]
Northern Lights [Chicago]

This is a good week to catch up and get a bit further ahead of ourselves in terms of documentation and supply orders. Relative to the pace over the last month, it feels like a bit of a vacation, though. :-)

Nysernet Chicago 10GigE is up

2007-02-09T03:48:00.000Z

This really capped the day off. Level3 has done an excellent job running down patches between their equipment and Nysernet's local carrier in the past week. The Nysernet interface is up in Chicago, so we can begin their migration.

Internet2 Washington DC node in service

2007-02-09T03:43:00.000Z

The old Indianapolis T640 has officially been redeployed as the Washington DC Internet2 node at Level3. To facilitate the change, the Internet2 Chicago to New York OC-192 circuit was split in Washington DC and the new router was inserted in the middle. The physical change on the transport equipment took about 30 seconds. The router config took about 30-45 minutes, depending on the service.

MREN transitioned in Chicago

2007-02-08T02:30:00.000Z

After a few days of wrestling with some faulty jumper cables between the Infinera and MREN Force10 switch at 710 North Lakeshore Drive, I was finally able to transition MREN over to the new router in Level3. Qwest is removing that PIC for us tomorrow and overnighting it to Washington DC for the install crew to place in the router.

Pittsburgh Ciena installed

2007-02-07T02:16:00.000Z

Tom and John completed the Ciena install in our Pittsburgh node today. It was a bit tight with Tom's plane being cancelled early yesterday, so they didn't get to work until Monday afternoon.

To top things off, Level3 called us late this afternoon to point out that our techs had left a door open at the POP. I got a hold of Tom and John just as they were getting to the airport, so they circled the terminals until I got verification from Level3 on the door and last scanned badge. Our guys turned back to the POP and verified that the door was indeed closed, but likely missed their flight. Someone owes them a few beers.

Starlight 10GigE migrated

2007-02-06T04:47:00.000Z

Just a quick FYI because I'm dog-tired and sick of sitting in front of a computer today...the Starlight 10GigE VLANs were all migrated over to the new Internet2 router in Chicago. We originally planned to also migrate MREN, but the new circuit is still taking a moderate amount of errors. We believe the issue is isolated to some faulty patch cords at 710 North Lakeshore Drive. Jake Sallee and I will continue to troubleshoot in the morning.

Chicago Metro DWDM up

2007-02-03T15:29:00.000Z

Jay and I finished the Chicago metro DWDM install around 5PM on Friday. Right now we're transporting the following connections between 710 North Lakeshore Drive and 600 W. Chicago

T640 to Legacy Abilene Chicago router at Qwest
MREN 10GigE
Starlight 10GigE
WISCnet/Merit Transitional 10GigE

Jake Sallee was a big help in getting all the fibers moved over at 710 North Lakeshore Drive.

We had planned to migrate all the Starlight and MREN VLANs over to the new network, but the MREN interconnect was taking a few errors on Thursday night. That, combined with the Indianapolis T640 router swap-out convinced me to postpone it. We're scheduled to make that move on Monday night.

Philadelphia to New York Dynamic Services OC-192 is up

2007-02-01T01:19:00.000Z

I received a very welcome phone call from John Graham today informing me that the Philadelphia to New York dynamic services lambda on the Ciena network was turned up and the Philadelphia node was reachable over the in-band management.

Pittsburgh and DC installs next week

2007-02-01T01:12:00.000Z

Next week will see a few installs in Pittsburgh and Washington DC. John Graham and Tom Johnson will be heading to Pittsburgh on Monday morning (well after the Colts have crushed the Bears in the Super Bowl!) to get the Ciena chassis and support equipment installed.

Following Pittsburgh, John and Tom will head to Washington DC to meet up with Andrew Lee, who will assist on the DC install. The Indianapolis T640 that's being pulled this week will arrive Wednesday morning and we should be able to insert it into the current New York to Chicago backbone circuit on Thursday evening. That will allow the migration of the following connectors:

NGIX-East
MAX
OARNET (from Indianapolis)

Ciena will be arriving in DC and Pittsburgh the following week (2/12) to perform the Test and Turnup of the CoreDirector CI at each site. By week's end, we should have the Ciena backbone up between Pittsburgh and New York. Once that's complete, both PSC and Drexel will be able to migrate to the DC router, as soon as they complete their installs into the I2 suite in Pittsburgh and Philadelphia, respectively.

Indianapolis router swap

2007-02-01T01:09:00.000Z

With the loaner T640 router that we shipped from New York safely in Indianapolis, it's time to put it back in the network. We'll be swapping it out with the Indianapolis T640 router, so we can send it off to Washington DC. The loaner needs to go back in mid-March, which works well for migrating the networks off. A few of them (the Indiana Gigapop and the University of Louisville) are relying on the DC-Atlanta-Chicago Level3 backbone to be handed over on March 3rd.

Chicago Infinera ring Day 1

2007-02-01T01:04:00.000Z

Jay and I completed the Infinera install at 710 North Lakeshore Drive, but we have some config work to do in the morning. There was some confusion over the bulk panel from our racks to the fiber meet-me area and I thought we were getting SC-style connectors. Instead, it's LC. We don't have much call for LC-LC connectors, so we don't normally keep them in stock. Jake Sallee has offered to loan us some of their fibers. We'll either swap them out later, or replenish Jake's stock next week.

Tomorrow we're off to 600 W. Chicago to turn up the Infinera ring. We can light one side during the day. The other side currently carries the new Internet2 network to legacy Abilene "network glue", so we've got a maintenance window to work on that in the evening. I'm also planning to migrate the Starlight and MREN VLANs over to the new Internet2 router so we can yank their PICs from the legacy router for use in DC next week.

There was a slight mishap in the POP involving a pile of squishy pink foam and Jay's glasses. The poor man can't see very well tonight, but we have some electrical tape! :-)

Roadmap for the week

2007-01-30T02:58:00.000Z

We've all recovered from the New York T640 swap and are focusing on the next set of installs. The Juniper loaner chassis we recovered from New York is on its way to Indianapolis where it will be swapped into the Indianapolis T640 on Thursday evening. At the same time, Jay and I will be heading to Chicago this Wednesday morning to install the metro DWDM equipment at 710 North Lakeshore. We expect to be bringing up the following networks while in Chicago:

MREN
WISCnet/Merit Transition
Starlight
Equinix
HOPI Internet2 L3
HOPI L2 backbone to Washington DC

I'm a bit excited to get out of the office. I've been pushing e-mails, spreadsheets, and project plans around so much in the past few weeks, I'm nearing my saturation point. That's always a good time for some DC power exposure. :-)

HOPI New York to Washington DC Circuit

2007-01-27T03:13:00.000Z

Thanks to the assistance of Chris Tracy and company in Washington DC and Matt Zekauskus who ran jumpers in New York about 2 months ago, the L2 circuit between New York and DC is back in service. This had been blocking on completion of a bulk tie cable between the Internet2 suite and the MAX suite in McLean.

We're going to Chicago next week and hope to get the DC to Chicago circuit up by mid-week.

New York swap complete

2007-01-26T04:29:00.000Z

The T640 router swap on the 24th floor is complete. Most connectivity was restored at 2215 EST, but the juniper syntax for overwriting a configuration threw me for a loop. It really just merged the two configurations.

The Surfnet OC-48 didn't come back up. It's showing RDI path errors, but we've seen that before during extended outages. The configuration looks good from our side.

The MCIT/ENERGI connection is still unreachable. They've been unreachable ever since we moved to the different T640 interface. I had been operating under the impression that it was a MAC-address problem because it's a TCC connect and the engineer setting it up 2 years ago asked me for the MAC address of our switch. I made sure to have the engineers put the OLD MANLAN 10G interface on the T640 end of the interconnect, but it's still down. Will need to troubleshoot more with the carrier in the morning.

T640 swap commencing

2007-01-26T01:18:00.000Z

The engineers in Chicago have gone dark as they move to the far side of the moon. We hope to re-establish connectivity soon. Farewell New York engineers and god-speed.

New York Router getting settled in on the 24th floor

2007-01-25T23:27:00.000Z

It brings me great pleasure to bring you the news that the T640 from the Qwest space on the 14th floor of 32 Avenue of the Americas has made its way up 10 flights of stairs on the backs of Jay Duncan, Andrew Lee and Hans Addleman. The oxygen tanks were empty and the Sherpa made its way back home by the time they reached the "summit".

Seriously, it's been one of the crazier days in my career here at the GRNOC. Not one that I care to repeat, but through the perseverance of some very dedicated people, the router is now sitting next to the Juniper loaner on the 24th floor just waiting to make a nest in the Nysernet space at 8PM EST.

We had a bit of trouble getting the Abilene Washington DC backbone circuit swung up to the 24th floor, but it managed to come up about 10 minutes ago.

Router swap starts in 1 hour and 27 minutes. Stay tuned. We're all heading out to refuel.

HOPI Interface migrated in New York

2007-01-22T04:53:00.000Z

I migrated the HOPI interface to the new T640 in New York. Took a lot of typing to move around all the MPLS tunnels that were CCC'd across the Chicago and DC HOPI nodes, but I'm pretty sure I got it all.

All that's left on New York is MAGPI. We're aiming to take the old New York router out on Thursday this week. It'll spend about an hour out of the network and take a ride up the freight elevator in 32 Avenue of the Americas to be swapped in place of the current new T640 on the 24th floor. That router is a loaner from Juniper. Rather than leave it in New York, we're going to bring it to Indianapolis where it can help hold things down until a few connectors are ready to migrate off. The timing of the move is somewhat dependent on when we can get the freight elevator operators to work. We'll push it as late as possible.

I'm taking bets on how fast we can swap out the loaner router with the older T640. The power's already there. Turn it down, swap the router out, move interfaces around and replace the config. I'm guessing 20 minutes. :-)

MANLAN migrated in New York

2007-01-22T02:07:00.000Z

The MANLAN 10GigE circuit has been migrated to the T640 in New York. This was essentially a VLAN move on the MANLAN switch, and Layer3 configuration on the Internet2 T640. All networks are back online except for MCIT/ENERGI, though I can find nothing wrong with the configuration. They've had issues with their network going offline during interruptions of service, so I've asked our NOC to get in touch with them to check their side.

I may still yet tackle the HOPI circuit tonight. Layer2 is already up, but there's a lot of router config. My head has been stuck in transitional issues since noon, so I think I'm going to take a break and watch the Colts like everyone else in Indiana.

MAGPI Layer2 is up

2007-01-19T19:16:00.000Z

Thanks for some fancy footwork by the MAGPI folks, we were able to get the link up between their router and the New York T640. We're working on configuring v4 BGP/MSDP/IPv6, etc.

MAGPI circuit will need to wait until tomorrow...

2007-01-18T22:05:00.000Z

The MAGPI wave is still an open issue. The U Penn folks are doing some troubleshooting with their local fiber provider tomorrow morning to find out why light isn't getting to our suite. Apparently there's a splice point through the second floor on its way from the 9th floor to us on the 3rd floor. The provider seems to think that may be where the trouble is. Hope to hear good things tomorrow morning.

I'm tentatively going ahead and scheduling the New York chassis swapout for the end of next week. We've gotta get that going so we can put it in Indianapolis to free that chassis up for Washington DC. Before you know it, Seattle will be affected if we don't get this done. See? :-)

The Ciena of Brotherly Love

2007-01-18T22:02:00.000Z

I heard good things from Andrew a few hours ago as he was packing up. The Ciena chassis is installed in Philly and all is well. Ciena is currently scheduled to do their test and turnup work the week of January 29th.

MAGPI wave troubleshooting

2007-01-18T15:42:00.000Z

We're working on getting the MAGPI wave up, but we're not seeing light from MAGPI in Philly. MAGPI is working on contacting their fiber vendors to verify light levels along the path. Hope to get some word soon.

The value of electrical switching on a DWDM system

2007-01-18T09:14:00.000Z

Thinking back on this, this was a fairly obvious observation, but I thought I'd mention something that I find to be very cool. I knew the theory of why this was so cool, but it was driven home by a real-world example that I ran through early this morning.

Level3 provisioned the MAGPI wave to show up on the second channel in the first bundle of 10 lambdas in New York, but on the first channel in Philadelphia. At first, I was a bit worried because I had some engineers pre-wire this particular lambda to the 3rd channel on the DWDM plan in New York. But then I realized I was thinking like a traditional DWDM groupie from the 90s. So, I took off my flannel shirt, turned down the Soundgarden and really got to thinking from a slightly less flanneled, yet still Pearl Jam'ed 2007 perspective. :-)

In a conventional DWDM system, a signal that's placed on a transponder card (translating your router's ethernet or SONET signal into something that can be multiplexed with other signals) is hard-wired to a particular DWDM lambda. Changing that lambda required changing a transponder card out to a different model- for those systems that were around before tunable DWDM optics- or at least changing the fiber run between the mux-demux card and a tunable card (for those later systems that do have tunable optics) The whole theory of having tunable optics really gets stuck on having to fix that jumper to the mux/demux card and you lose some of the dynamicism that's so heavily touted along with it.

So, back to the Infineras. Since the signal is electrically regenerated at each node, switching the lambdas around (and switching the client signals around) is a piece of cake. On the New York Internet2-owned DWDM ring, I have a client signal coming into port 2 from Level3 at 111 8th Avenue. I wired the 32 Avenue of the Americas node's client interface up to port 3. Instead of having to re-wire the 32AoA side to dump the signal out on port 2, I can simply create the circuit between ports 2 and ports 3 and be done with it. There's essentially a complete decoupling of port 2 and channel 2. The two have nothing to do with each other.

I knew you could do this on the lambdas that crossed the backplane between different fiber directions, and I suppose I figured I would be able to do it within a particular card, but it just dawned on me just how powerful that feature really is from a real-world operational standpoint. This saves us from having to move jumpers and allows us to bring circuits up really fast. Theory is great, but when you start using it, the value just pops off the screen.

Philly update

2007-01-17T22:49:00.000Z

Andrew and John report that things are moving along in Philadelphia. There's an odd problem with the rack not being quite wide enough and throwing off some of our rack ear extenders. We're overnighting some 5RU extenders, which is unfortunate because they'll hang into dead space and make it difficult to put any future 23" equipment there. (But let's be honest, whatever goes in next will likely be 19" anyway). They were also missing clear heat-shrink from their kit of materials, so that got overnighted as well.

I'm working with Level3 on getting FDP assignments for the lambdas coming out of Philly (and actually turning up the MAGPI circuit that's been ready to go since Tuesday). Hope to have that info by tomorrow morning so they can run patches and roll the fibers if need be.

Philly Ciena install underway

2007-01-17T13:50:00.000Z

Andrew and John are relaxing comfortably in a Northwest Airlines coach seat right now, dreaming of the Philadelphia install that will happen in a few hours. They should be on-site by 11AM and start knocking through the install shortly thereafter. Unfortunately, with these installs, there's no "EUREKA!!" moment when the network comes up since we're having Ciena slot the cards and turn up the node. In general the techs will have a POTS line to install, but who gets excited about modems anymore?

We're hoping to get Ciena out within the next few weeks to complete their part.

Philly rack swap complete

2007-01-11T17:04:00.000Z

The Philadelphia delays are even more slight than I thought. There was an e-mail in my inbox yesterday that I hadn't gotten to yet. The Philly techs replaced the rack immediately with the appropriate 23" rack and mounted all our hardware. The power, bulk panels and multifunction panel are all installed. It's ready for us to visit and get the Ciena installed.

Slight setback in Philly

2007-01-11T05:13:00.000Z

I got a rather confusing call from a Level3 POP technician in Philadelphia today. He was installing the Sentry power controllers and said he didn't have the right rack years. The controllers ship with 19" ears, but we bought a bunch of 23" versions. I was certain that we included the 23" ears in that shipment. It turns out that the rack delivered in Philadelphia was a 19" rack instead of 23". That isn't going to hold the Ciena, so Level3 is swapping it out. My understanding is that this should happen before this Friday, but it's tightening up the MFP and power controller install. Assuming it happens within that timeframe, no harm, no foul.

Ciena slotting and On-Center

2007-01-11T03:49:00.000Z

After a bit more discussion with Ciena, we've decided not to pre-slot the CoreDirector CI cards before we leave the install. The Ciena Test & Turnup technicians normally handle that and it's best to leave it to them. They'll be checking for bent pins and those sorts of hardware defects, so they'd want to remove the cards anyway. We'll just leave dangling fibers (*gasp*) and they'll route them for us.

Word came down from our system engineering team that the Ciena EMS was successfully installed on our management box. The software isn't the most intuitive piece of equipment to install and Ciena was kind enough to arrange for a technician to come out and help out twiddle the appropriate bits on the system to make it all gel. Both New York and Chicago and already visible.

BTW, Ciena has officially certified both Chicago and New York, as of yesterday. We have a bit of test data to look over, but that will be more of a formality.

Eat more fiber

2007-01-10T03:36:00.000Z

There are a lot of questions about all the different fiber panels on the new network. In general, there are three different types of panels in use:

Bulk panel to Level3
These panels terminate the bulk fiber between the Internet2 suite and the Level3 centralized fiber termination block, known as the OSX. This bulk cable is used to carry client signals between equipment in our suite and the tributary cards on the Infinera transport gear. In router locations we have 96-strands (48-pair) of fiber going in each direction out of the city. So, in a three-way router site, we have three 96 strand cables going to the Level3 OSX. At smaller Ciena-only sites, we have 48 strands (24 pair) going in each direction.

Each 96 or 48 strand bulk cable is terminated across 4 or 2 single RU ADC fiber panels. These panels terminate 24 strands of the cable into SC-connectors. The panels are divided into two sides, with the left side of the panel aiming fibers out the left side of the rack, and the right side aiming fibers out the right side of the rack. (This becomes important when determining fiber lengths.) There are two rows of 6 strands on each side of the FDP. Strands 1-12 are on the left side and 13-24 are on the right side. The ports are numbered from left to right, with 1-6 on the top of the left side and 7-12 on the bottom of the left side. Level3's standards are to use ports that are on top of each other, rather than sequential numbered ports that are side by side. Hence, their circuits will terminate on ports 1 and 7, 2 and 8, 3 and 9, and so on. The right side of the FDP is numbered similarly, with the first port being 13.

Multifunction Panel
These panels are very simplistic 1 or 2 RU panels that have modular-style connectors. In router sites where we're obtaining a Level3 High-Speed IP (HSIP) ethernet drop or a coaxial timing connection, we have modules to support that. The rest of the connectors are generally SC-style to terminate customer fibers that land in our suite. We've issued LOAs to customers to allow them to land fibers within our suite. No customer fiber is to connect directly into our gear, aside from those within our suites. The customer fiber should land in the MFP on the back of the panel in the designated ports. Internet2 is responsible for the fiber patch between the MFP and the various pieces of equipment within the suite.

Ciena Fiber Trays
The Ciena interfaces often require simplex fibers to be run out different sides of the card. It's acceptable to run simplex fibers within a particular rack, so at Ciena-only sites where the Bulk Panels to Level3 are within the same rack, it's an acceptable fiber run. For router sites where the bulk FDPs are often in another rack, we decided against running the more fragile and less-available simplex fibers through the overhead fiber infrastructure and are placing a 1RU fiber distribution tray on top of the Ciena. This tray converts the LC-style connectors on the Ciena to SC-style, so we can maintain stock of the more common duplex SC-SC fibers.

Miscellaneous FDPs
There are some additional panels that are used in some of the 3rd party sites that fall outside of the norm. They are:

NEWY32AOA - bulk FDP to Fiber Meet Me Room (FMMR) - 48 strands
CHIC710NLSD - bulk FDP to 710NLSD Fiber Meet Me Area
WASH - bulk FDP to MAX suite on the same floor - 72 strands

Ciena turnup

2007-01-10T03:20:00.000Z

We've been discussing the Ciena node turnups a bit lately, with the majority of the coaching coming from John Graham. Recall that there are two types of sites that have a Ciena in them. Router-node sites, where the Ciena install will be timed to coincide with the router install, and Ciena-only sites that usually just have a single rack of equipment. Each turnup has three main phases

Installation
This is done by an Indiana University install team of at least two people. It will include getting the chassis into the racks, and terminating the power on the Ciena. Getting the chassis in the racks at a remote site can be challenging, given the weight issues. The Level3 standard racks have a bit of extra space at the bottom, between the lowest RU and the kickplate. That likely means that the screw holes won't line up properly with the Ciena rack ears. Getting those to go up with a two-man team can be a bit dodgy at times, so we're looking into some sort of front and back jack system that we can ship with the install kits to give that small 1/8" bump that it's hard to do without the Incredible Hulk as part of the install team.

Engineers will install the observatory and control PCs, the small racklan switch and the out-of-band dial-up router at this time, but there will be no data plane connectivity because we aren't turning the Ciena up. We'll pre-run the backbone fibers from the Level3 bulk FDP to the Ciena, but they will be clipped to the 10G cards and not inserted.

Test and Turnup
Turnup and configuration of the node is left to Ciena. They will arrive shortly after we've handled the chassis install and power and turnup all the cards. After each card is slotted, they'll run them through a series of traffic tests that would make even the most cautious of us feel somewhat like a network cowboy. Ciena has agreed to have their installers plug the pre-run backbone fibers into the OC-192 backbone cards when they've completed testing.

Node Certification
Once the card is installed, and an adjacency has been established between the new Ciena and an already-activated Ciena, it will become our playland. But first, John needs to do some quick checks of the system to make sure it's good to handle production traffic. A lot of that also has to do with documentation and monitoring. We don't want to put anyone on the Cienas without proper back-end support.

And there you have it. Ideally this will be a two-week process (one week spent getting it installed- only one day of which will actually be used to handle the install) with the next week for the test and turnup.

Site-by-site update

2007-01-10T03:05:00.000Z

Things have been moving so quickly this past week that I haven't had much time to sit down and type up an update. Apologies all around and I promise life will get better!

Chicago
We initially thought we'd get the DWDM ring installed in Chicago this week. The power work was complete last week, but the bulk fiber panel from our rack to the 710 North Lakeshore Fiber Meet Me Area didn't make it in time. The last estimate I have is a week or two, but we're working to get that tightened up. We have the option of doing temporary long jumpers, but in our business, temporary things tend to go undocumented and become a part of the permanent landscape. We'll avoid it as long as we can. It won't be a picnic to not have that up until the end of January, but it won't really push things out too much since one of the connectors won't be ready to move off the old Chicago router until the end of the month. Hopefully we can get that date pulled in.

Philadelphia
The suite was accepted and the power installs are on track to complete this Thursday. The MFP is also being installed by Thursday and we expect the MAGPI cross-connect from their metro dark fiber provider to be terminated on that panel on Friday. That leads us to New York....

New York
Once MAGPI is out of New York, we can remove that router from the Qwest floor and move it up to the new space. We'll swap out that Juniper loaner that's been handling some of the new connections already and ship it to Indianapolis to pull its weight for us there.

After a bit of MTU-wrangling, the new NOX R&E peering is fully transitioned over to the new router and they're going to turn down their OC-48 on Thursday.

I talked with Jim Shaffer, who's in New York this week helping with the ESnet install at 32AoA. Things went well with ESnet, apparently. They sound like they've accomplished their install in two days time. Kudos to ESnet and welcome to the new Internet2 suites. I wish I left a doormat for you....

Pittsburgh
We got back the photos for our remote suite acceptance and found enough issues to reject it for now. There's a call tomorrow with Level3 to go over the bits that we think need addressing and hopefully it'll get taken care of within the next few weeks. We do have Cienas to install, after all

Cleveland
We heard last Friday that Cleveland was actually completed on December 21st. After looking at accepting the suite remotely, we decided it would make more sense to do it in person. We're accepting all the router sites in person. There are two sites (Cleveland and Sunnyvale) that are multi-row Ciena-only sites. It's difficult to check things like inter-row signalling trays and other overhead infrastructure, so we'll be doing those two sites in person. Jay Duncan is enroute and will be on-site tomorrow afternoon to handle install.

Boston
It looks like Level3 has located a suite that recently came available to us, so we're going to go out on our own instead of colocate in the NOX suite. A big thanks to Leo Donnely for all his work in entertaining that option.

Back from the holidays

2007-01-03T02:31:00.000Z

We're still all recovering a bit from the past few weeks of mismatched work schedules, but people are trickling back into the office and wiping the end-of-the-year sleep out of their eyes.

Today was spent getting people back up to speed. A few highlights.

We're awaiting some metrics on the suite in Philadelphia so we can accept it.
We were notified that the Pittsburgh suite was completed today
Once we get Philly and Pittsburgh accepted, we'll get power and multifunction panels installed within a few days
MAGPI is close to being ready in Philadelphia. They need to procure one last jumper from their metro provider to our suite. Hope to get that done in the next week
NOX is going to switch over their primary traffic to their new 10G on Thursday and do a 1 week burn-in
The new Internet2 NOC website address is ready and a lot of the text has been changed to the new name. (But I won't tell you the address just yet!)
Ciena has shipped the Cleveland, Philadelphia and Pittsburgh chassis. We'll install shortly after the DC power is installed by Level3. Ciena is waiting on a new revision of their OS to hand over the devices in Chicago and New York.
The HOPI lambda between New York and DC is blocking on re-termination of the bulk cable between the I2 suite and MAX's suite in Washington DC.
Jake Sallee tells me that power should be ready at 710 North Lakeshore Dr. in Chicago by tomorrow. That likely means the DWDM metro gear will get installed next week.

There have been a few scenarios floated around to get a router to Washington DC. It's almost certainly going to be the Indianapolis T640, but we need to get the DC router installed before the Atlanta to Chicago Dedicated Wave System (DWS) becomes available. So, we can either move people off Indianapolis ahead of that, or we can put a new router in Indianapolis. The current thought is to get peers off New York within the next few weeks and move that Juniper loaner router to Indianapolis. We should know more in the next day or two whether that's going to be possible.

As Rob Vietzke said this morning, "it's time for the rubber to hit the pavement."

Making waves in December

2006-12-18T23:28:00.000Z

We placed orders for the next four waves we need to light up. Two are for the HOPI backbone (Chicago to DC to New York). One is for MAGPI via Philadelphia (which we're working on accepting the suite) and another is for Nysernet via Cleveland. Level3 provisioned them almost immediately and is working on a few fiber patches in DC. In all cases, we have some final patches to run on the ends, like with any circuit turnup, but the Infinera backbone portion is more or less complete.

Caren worked up a checklist of items for the Philadelhia remote suite acceptance. Looks good and very thorough. Hope to have that checked off shortly. The MAGPI circuit is still a work in progress. Looks like there's a final patch between MAGPI's dark fiber provider and Level3 that didn't get on the radar screen until Friday. Hopefully we can make that happen in the near term.

The colocation at 710NLSD sounds like it's getting close. I believe the power was supposed to be installed today, though I don't have confirmation yet. If that happens, John Graham and I may head up to Chicago on Wednesday and Thursday to turn that up.

It's a good time of year to do some documentation catch up. I've spent most of the day working on translating some of our install notes into more formal documentation.

Philly colo delivered

2006-12-15T03:12:00.000Z

Word came down today that the Philadelphia colocation suite is ready for acceptance. In the interest of time, we're going to have Level3 run through a checklist of items and provide photographs of the suite to accept it remotely. Our hope is that for these smaller Ciena-only sites, we won't need to visit them all in advance of install. This is our first go around with that, so we'll see how well it works out. It will probably be a day or two (or three) while we go back and forth on that.

The multifunction panel that I2's customer circuits will terminate into is on-site and we'll get Level3 remote hands to install it shortly after the acceptance. It's needed ASAP to terminate two fiber cross-connects from MAGPI that should drop in the next several days. Once that's up, we can bring MAGPI up on the new network.

Ciena cabling, schedules and meetings

2006-12-13T04:26:00.000Z

Even though I was hoping everyone would slip into a three day coma after the I2MM last week, it turns out, the world keeps churning and we still have a lot of things to keep plugging away out. No rest for the weary!

Half of our prayers were answered this week with the arrival of a new engineer at IU. John Graham comes to us by way of UKLight in Great Britain. Fortunately, he's very well versed in SONET- specifically Ciena equipment. We're very fortunate that John decided to join up and make the move to Bloomington.

The Ciena buildout is just starting to get into action with our first few boxes shipped and installed over the past few weeks. John's arrival is perfect timing and he'll be running the Ciena show for the next several months. That frees up some cycles amongst the rest of us to keep our attention on the IP network transition and the Layer1 issues. John has already been instrumental in pointing out that the CoreDirectors are quite complicated beasts to cable up and that we should pre-wire the front panels to a separate fiber tray. It's that kind of real-world experience that will make the Ciena network a success.

Lots of meetings coming up. Level3 is bringing a whole slew of folks to Indianapolis on Wednesday to review our final operating procedures. We'll get to meet our dedicated support engineer that will be answering the phone when we call during business hours for Level3 technical assistance. There's been a lot of informality so far in handling these first few wave turnups, so we'll finalize the procedures that will enable us to move even faster.

Thursday is a meeting to discuss implementation of the layer2 services. There's some good documentation floating around that we'll look over and discuss before presenting it to the community for input. I'm really interested to see what sorts of things people have planned for the Ciena network, as I'm sure is the rest of the team.

Next Monday is a meeting at Ciena to talk about control plane management and implementation. John Graham will be attending in person from IU, and Matt Davy and I will be joining via phone bridge.

The schedules are getting more focused as more information rolls in. Right now, it looks like Philadelphia and Pittsburgh installs will be the first or second week of January, though there are still a few niggling details that I expect to be sorted out.

Right now, we're hoping that MAGPI's 10G wave to New York from Philadelphia will be ready sometime next week. The suite sounds on track for delivery tomorrow and we have to get it accepted and run some fibers first. Hope to have that all taken care of by Friday.

It looks like the 710 North Lakeshore facility in Chicago is having a few issues with the overhead ladder rack not being ready in time, so the last I've heard, the availability of power has moved to next week. That's our top priority now since it's needed to move the Indiana Gigapop off the Indianapolis router. Once we free up Indy, we can ship it to Washington and the next phase of transition starts up.

That's all for now. Lots going on. Should have more to report shortly

More Behind the Scenes from the I2MM

2006-12-11T21:51:00.000Z

Laurie Kirchmeier graciously put together some more "behind the scenes" notes on the I2MM demonstration last week. He was manning the iHDTV video encoders and decoders behind the big black curtain. He's also supplied some photos of the equipment setup behind the stage:

Getting the live uncompressed HD video from the test camera set up in the colo space the day before the plenary address was a spine-tingling moment. We had been fighting with flaky iperf measurement tests for a couple of hours and had gone to the length of deploying a cakebox measurement box on the Juniper router located in the main telephone closet in the bowels of the convention center. When we actually ran a live video test, it turned out to be flawless with no packet loss and a wonderful view of the moving second hand of an analog alarm clock and a small computer monitor.

The morning of the event meant an early start for those of us in Chicago because of the time difference. The NYC film crew started setting up at 7.30am EST and by 8am we had color bars being transmitted over the iHDTV connection and audio testing began.

The first live video from the SteadyCam was impressive and the rehearsals looked and sounded great.

With an hour to go, Mike La Haye and myself got set behind the stage and gave countdown times to the NYC side via IM. We were too pleased when the camera kept being switched off as the software needs to stabilize with a running video stream and after the 20mins to go message was sent out things got a little tense. whilst trying to reset one of the ethernet interfaces on the NYC side via Remote DeskTop, I managed to reset the wrong interface and promptly lost my Remote Desktop connection to the machine and had to request that the sender machine be rebooted. After reconnecting and re-establishing the live iHDTV stream we watched nervously as the camera crew and Tim performed another reheasal. As they were finishing the rehearsal, I realized that Doug was making his last point of his presentation and instead of 10 mins to go, we were down to 1 or 2 mins before he introduced Tim and we needed to go live. I furiously sent impassioned messages over IM to get them back to the sttartign position for the broadcast and tried to indicate the urgency and Mike was on his cell phone talking with Aliza who was relaying the latest information to the crew and to Tim.

To raise our blood pressure further, the cameraman disconnected the camera and then we listened as Doug started his introduction to the broadcast. Mike was now speaking very calmly and firmly into his cell phone telling them to get ready and get the camera rolling again - I was watching the receiver console and the video monitor and finally we got live video and audio. We both looked at each other and them realized that the HD projector needed to be flipped on to the screen. This required the projectionist to turn off the standard slides projector and flip up the lens cap (a piece of cardboard hung over the front of the lens) on the HD projector. As switch was made and the HD video appeared on screen to the audience, I realized that we were not getting full 1080i video and the sender and receiver nodes were not properly synched. This resulted in lines appearing across the screen and the audio was slightly garbled. The solution was to reset the receiver software by hitting and re-starting the app. I reset the software and the video and audio glitches remained. Help, the reset normally just works! I reset the software a second time and still the video and audio were not perfect. By now Tim was 30 secs into his presentation and he and Doug made some comments back and forth. During one of Doug's comments, I tried one more reset and magically the video and audio stream problems disappeared and the audience was treated to the remaining 9 mins of Tim's presentation shown in uncompressed 1080i video and 48KHz audio.
Mike and I were out of our chairs watching the projected video from the rear and I remember nervously counting down the minutes in my brain until the presentation was over.

it was a great sense of relief to have have the event over and a tremendous sense of accomplishment to have pulled off an awsome demonstration over the brand new New network.

Over the next couple of days I was happily surprised by the comments for the attendees and to realize how much of an impression the iHDTV presentation over the new Internet2 Network had made with everyone. It certainly was a great team effort.

Infinera 100G ethernet demo box

2006-12-06T17:01:00.000Z

Went to the POP this morning to troubleshoot some trouble on the 10th lambda that cropped up yesterday evening. It turns out it was a bad jumper on the run from the Level3 bulk panel in our New York suite to our Infinera metro equipment. Once we got that cleaned up, the circuit normalized itself. Our thanks go out to the Level3 team that ran with this (Eli, Nate, Ryan and probably some others behind the scenes)

Here's a shot of the box in our rack with 10 "greened-up" interfaces to New York. The technician in the background is Steve Armstrong, from Infinera, verifying connectivity on his laptop.

Behind the scenes of the iHDTV broadcast from New York

2006-12-06T12:22:00.000Z

Tim Lance, president and Chairman of Nysernet, wrote up an excellent narrative of his experiences on the New York end of the live HD stream during yesterday's plenary. Many thanks to Tim for putting this together:

After the 32 AofA tour during Doug's opening plenary, I flew to Chicago and that evening heard comments about how stunning the video was with the sound almost as good. But an operational collocation facility is about as environmentally friendly a broadcast facility as a running shower, so the clarity represented real magic by everyone involved with the network connection and the iHDTV software, with built in escapes (as were used at the start when the interlacing got out of synch. Internet2 folks had worked on the sound and video transmission the week leading up to the live tour, with conversations between them and the TV crew, while I worked on what should be said with help from many.

The TV crew had an equally daunting task, masking the roar of air conditioning and equipment, figuring our how to light the path of the tour (not creating terrible shadowing or hot spots). We were in the colo space early on December 5 as cage walls came down and we did dry runs for sound and video, plus two dress rehearsals, and made eight or nine different placements and settings of lights before the crew was satisfied. The moving picture required manipulating a very large and heavy steady-cam, anchored to a flack-vest like contraption on the cameraman who backed down aisles almost too narrow for all the gear. The wide angle lens made the aisles seem wider than they were.

This was a really gutsy move by Internet2. They started planning this live demo just a week or two before the Member Meeting, and before the first segment of the network was up. Then they decided to do the first test of the power of the new network live and in public with a tour the most inhospitable (from a TV perspective) environment around. Perhaps it's not surprising that it all came off so well, given all the talented people working on it, but it surely demonstrated that we've entered a new era of network capabilities.

From one to nine in 31 minutes...

2006-12-05T19:56:00.000Z

I started talking with Level3 at 14:08 about changing the framing on the CHIC-NEWY backbone OC-192 to 10GigE and bringing up the other 9 lambdas on that path. This IM exchange with Rob Vietzke says it all:

[14:39] NOC - Chris Robb: 9 lambdas up. took 31 minutes
[14:39] rpvietzke: cool
[14:40] NOC - Chris Robb: It includes the amount of time to get a tech to check it
[14:40] NOC - Chris Robb: If that weren't part of it, it would have taken about 5 minutes

I'm smiling. :-) <---Redundant emoticon used purely for illustrative purposes

The network is up, now let's tear it down!

2006-12-05T17:42:00.000Z

Now that the network has been up and running for a few days, we're going to tear it down. :-)

As part of the Internet2 Member Meeting Plenary session, I2 CEO Doug Van Houweling and Nysernet CEO Tim Lance participated in an uncompressed HDTV video presentation from the New York POP. The 1.5Gbps HD stream traversed the new network. Steven Wallace took some photographs of the presentation and quickly stitched together a panoramic shot of the event.

Now that the network has run through it's paces, we're going to tear down the OC-192 between New York and Chicago to be utilized for a demonstration of an experimental 100Gbps Infinera chassis between Chicago and New York. The OC-192 will be reframed as a 10GigE and the other 9 10GigEs on the path will be lit between cities.

I'm heading to the POP very shortly to run the jumpers for the Infinera box.

BTW, Welcome Slashdot Readers (TM). :-)

Would you like to see where everyone goes?

2006-12-04T20:54:00.000Z

I'm happy to share with you a map we've been using internally to keep track of the entry and destination points of all the Internet2 connectors. This shows lambda routing for connections to both the IP and grooming networks.

I lovingly call it "the subway map" since it took a PhD in diagramology to keep the lines straight.

The ugly face of circuit tracking, Part 2

2006-12-03T03:18:00.000Z

I posted before the first install about some of the challenges in nomenclature, metadata storage, and mapping abilities of having so many fiber patches and logical portions to keep track of in an optical network. I also talked about the impending system to keep track of it all. I'm happy to say that the system appears to be very well fleshed out. I saw an optical path illustrated on one of our other optical networks we manage, ILight. It's just simply stunning.

So I sat down tonight and spent an hour diagraming up the path of the first backbone circuit between Chicago and New York for the database guys to work on getting into the database. I suspect the different numbering schemes of the various fiber distribution panels (FDPs) is going to trip them up, but this is a good dry run.

So, what's good enough for our database programmers is good enough for you. Here's the diagram I sent him (with some key bits of information blurred out to protect both their nakedness and true values) along with the explanation I sent that described the data path. It's a bit more detail on just how challenging this sort of thing is to track on a large scale. Everyone I've talked with, aside from major carriers, does this sort of thing on spreadsheets spread across several engineers laptops. Getting this information into a database represents a quantum leap in the way we handle our POP metadata. I'm quite proud to work with colleagues that can think this hard about a problem and come up with such an elegant solution. I would suggest that you click on the diagram and open it in another window while reading along:

Starting in Chicago....

The T640 connects to a Level3 single RU panel in the position indicated. I've indicated the connector types on each end throughout. It terminates 24 simplex SMF fibers. They come prelabled from the manufacturer in the following manner: the chassis is split in half with 12 strands on the left and the next 12 on the right. The top row on the left starts with port 1 and increases to the right until it gets to the middle (so, 1-6). Then the numbers pick up under that (7-12). The right side is similar with number 13 starting on the top left port of the right side and increasing to 18. Under that is 19 through 24. Level3's standard is to use ports on top of each other rather than the ports next to each other. That means, instead of seeing port assignments 1 &2, you'll see 1 and 7.

That bulk panel runs to some Level3 panel somewhere and into their transport cloud. Eventually it ends up on another bulk panel in New York at the ports indicated.

That New York bulk panel is at 111 8th Avenue, but we have our router at 32 Avenue of the Americas (32AoA). To get this signal between buildings Internet2 purchased Infinera metro DWDM boxes that we manage and Level3 metro fiber between buildings. The client signal from Level3's bulk panel (framed at OC-192) drops out of the bulk panel into a client port on the Infinera (the DLM in slot X of the chassis, the TAM card in slot X of the DLM, and the TOM in port X of the TAM) From there, it is muxed onto a DWDM network between 111 8th Avenue and 32AoA. So, on the diagram from BMM to BMM on the two New York Infineras is a DWDM-capable path that can carry up to 40 circuits.

We don't really have a separate naming convention for those circuits and I went with the same name as the Level3 circuit ID. It occurs to me that we could probably make up a better circuit ID for that portion of the circuit. Perhaps something like NEWY1118TH-NEWY32AOA-0001.

The Level3 metro fiber lands in the 111 8th Avenue suite on a Level3-provided multifunction panel. It has 4 "slots", each accepting a different type of fiber, copper, or coaxial termination model. It's essentially just a piece of sheet metal holding a bunch of couplers together. Our BMM connects to ports 1 and 2 on the first slot. The particular module in that slot has 6 SC fiber couplers in two rows, numbered from left to right (so, 1-3 on the first row and 4-6 on the second row)

From that MFP, Level3 has two dark fiber strands between 111 8th Ave and 32AofA. I didn't indicate in the diagram, but the fiber in port 1 of the 111 8th Ave MFP is connected to strand 11 (the top strand) on the run betwen buildings. The circuit ID listed for the dark fiber is Level3's.

The Level3 dark fiber terminates onto a patch panel in the fiber meet me room (FMMR) in 32AofA. From there, Internet2 is responsible for a 15m jumper over to Internet2's panel in the FMMR. I don't have the rack assignment for that in my notes, but I'll try and get it from Nysernet. The I2 panel in the FMMR terminates a 48-strand bulk cable that runs over to a Leviton FDP in our suite. The Leviton FDP spec sheet is attached. It's a 2RU box populated with four 12-strand modules. Best to just look at the photos here to get the numbering scheme:

http://www.pbase.com/i2net/image/70365530/original

The upper left is port 1, numbers increase to the right. 1-6 on the top row, 7-12 on the second row, 13-18 on the third row and 19-24 on the fourth row.

From the front of that panel, the composite metro DWDM signal goes into the Infinera BMM. From there, the Infinera client optics drop it to the T640 as a framed OC-192.

As you can tell, there's quite a bit of variation in the patch panels in the path and their numbering schemes. The database will need to accomodate that.

Here's the photo of our original whiteboard discussion of how these circuits are tracked:

http://photos1.blogger.com/blogger/4378/146/1600/IMG_0024.jpg

Recall that we were going to have one overall circuit ID for this entire end to end path. That hasn't really been specified yet in this discussion. I've been calling the circuit by the Level3 circuit ID name in all labels because we don't yet have the database ready to link disparate circuit IDs together into something someone would be able to figure out without the diagram I've drawn up.

I know the GLIF has been doing some work on this sort of thing and have heard a few folks talk about going through the motion of modeling portions of their network in Network Description Language. I honestly don't know what the underlying mechanism here is, but perhaps that's a good topic for another post from someone who can speak more definitively...

See you all in Chicago.

DNS is up

2006-12-02T09:40:00.000Z

It's somewhat troubling when minor events get us engineers excited. The simple act of having a name for an IP address is the modern equivalent to the Oklahoma Land Run. This is our name and this is how it's going to look for the next 7 years, dear readers:

clarus:~ luke$ traceroute www.mit.edu
traceroute to www.mit.edu (18.7.22.83), 64 hops max, 40 byte packets
1 129.79.9.254 (129.79.9.254) 0.855 ms 0.391 ms 0.265 ms
2 ibr.noc.iu.edu (149.166.2.52) 1.266 ms 1.338 ms 1.285 ms
3 149.165.254.229 (149.165.254.229) 1.321 ms 1.386 ms 1.255 ms
4 abilene-ul.indiana.gigapop.net (192.12.206.249) 5.579 ms 1.544 ms 1.490 ms
5 chinng-iplsng.abilene.ucaid.edu (198.32.8.76) 15.656 ms 5.183 ms 16.110 ms
6 ge-0-0-0-10.rtr.chic.net.internet2.edu (64.57.28.1) 23.546 ms 12.933 ms 5.502 ms
7 so-0-0-0-0.rtr.newy.net.internet2.edu (64.57.28.8) 33.855 ms 27.289 ms 27.285 ms
8 ge-0-1-0-10.nycmng.abilene.ucaid.edu (64.57.28.7) 27.282 ms 27.222 ms 27.265 ms
9 nox230gw1-po-9-1-nox-nox.nox.org (192.5.89.9) 32.445 ms 32.466 ms 32.438 ms
10 nox230gw1-peer-nox-mit-192-5-89-90.nox.org (192.5.89.90) 33.292 ms 33.278 ms 33.318 ms
11 w92-rtr-1-backbone.mit.edu (18.168.0.25) 32.759 ms 36.684 ms 32.633 ms
12 www.mit.edu (18.7.22.83) 33.024 ms 33.196 ms 33.066 ms

Please try to contain your excitement and save it for the RRD graphs. :-)

Take a deep breath...the next 7 months

2006-12-01T22:24:00.000Z

If you have an inquisitive mind, you're probably wondering how we'll transition between the two networks while still maintaining the day-to-day production connectivity the Abilene connectors have grown to love so much. I'm happy to share a set of 9 diagrams that show the network through it's various phases of evolution. I should add that this is the current plan and is subject to change as new dependencies crop up.

Phase 1 is complete! The router in Chicago is our old lab router from North Carolina and the router in New York is a loaner from Juniper:

Phase II will happen in January, once we transition networks off our Indianapolis router. That will free up a third "floating" router since Indianapolis doesn't get a router in the new network. (We're all a little sad in Indy to see our T640 go away, but these sorts of things happen) We'll have moved everyone off the old New York router, so we'll carry it up the 10 flights of stairs- or use the freight elevator, whichever is easier- to replace the temporary Juniper loaner router:

Phase 3 will see the completion of the next major round of Level3 routes in early March. By then, we'll have connectors transitioned off the old Chicago T640 and will have moved it to Atlanta:

Phase 4 will see the movement of the old Atlanta and Washington DC routers to Kansas City and Houston, respectively:

Phase 5 will see the turnup of the new Salt Lake City router after we move the old Kansas City router connectors off their node. This will pave the way to transit Denver networks:

Phase 6 will see the movement of the Houston node to Seattle:

Phase 7 brings a close to the turnup of the new backbone with the old Denver node moving to Los Angeles:

The final Phase brings a transition of networks off the Los Angeles and Sunnyvale nodes and their de-install:

Traffic moved to L3 Chicago-New York path

2006-12-01T21:47:00.000Z

I just adjusted the backbone metrics to prefer the Level3 backbone circuit between Chicago and New York. The NOC's weathermap is now up to date with the new link, thanks to our programmers. DNS should be in the system within minutes. Given those two last pieces, I adjusted the IGP metrics down on the new link to prefer it over the legacy Abilene circuit between Chicago and New York. Here's the first snapshot of the weathermap right after the move:

I'm also attaching a diagram of the network with the current set of backbone metrics. These sorts of diagrams will all end up on a static webpage and be updated as we move so you don't have to follow the blog. I hope to get that worked out over the weekend and live on Monday.

Nysernet completes transition to New York

2006-12-01T16:34:00.000Z

Got a note from Bill Owens this morning that they're transition to the new network went smoothly this morning. They're passing all traffic in New York via the new router and have turned down their old peering.

Shipping hiccups, I2MM showfloor, and Nysernet IPv6

2006-12-01T02:57:00.000Z

Rob Vietzke was in New York today finalizing some prep work for the Internet2 Member Meeting. We had hoped to have him run some fibers between the Level3 bulk panel toward Chicago and the Infinera, but the fibers we shipped to the wrong zip code. Apparently New York is so dense that it has exactly 5,768 zip codes. :-) That meant that the fibers didn't make it in time and we'll need to do remote hands tomorrow.

Ciena had some similar problems with their 10GigE cards that they sent for the installers over a week ago. Apparently they were off by one number on the zip code and they ended up 10 blocks away from where they were supposed to be. Oddly enough, they were delivered and signed for at this mythical address. Fortunately, the carrier was able to locate them and get them into the installer technicians' hands by the end of the day.

Had a bit of back and forth with Christian Todorov and Linda Winkler about the member meeting network. I'm attaching a diagram for fun, though the VLAN assignment has already changed and I now have the AS number of the showfloor. The legacy Abilene router will be connected to the network via it's existing MREN peering. The new Chicago router will have a layer2 cutthrough directly to the showfloor. So, by default the new router will be the default path to and from the showfloor (based on AS-path length). I'm not sure which way we want to go with that, but I think folks want to keep the legacy router as the primary link. So, I'll pad the AS toward the I2MM and let MEDs take over. We redistributed our IGP metrics into BGP MEDs. Since both routers will be advertising the same set of BGP routes, the showfloor will have two paths to Abilene connectors. Since the IGP metric on the new CHIC-NEWY circuit is still higher than the legacy Qwest Chicago to New York link, the only networks that will have a lower MED (and be preferred on the new router) will be those networks that are homed directly to one of the two new T640s.

Speaking of networks that are homed to the new network, Nysernet is planning to complete their transition to the new network at 0630 EST tomorrow morning. I just traded IMs with Bill Owens this evening. IPv6 is already transition and he's seeing the Kame turtle. IPv6 multicast works as well. We sourced some >1Gbps multicast streams from New York this afternoon and all went smoothly.

BTW, I'm told that the new weather map will be ready tomorrow. Reverse and forward DNS will happen tomorrow or Saturday. Our systems engineering guys are bringing up a new tool that auto-generates all our DNS off our network topology database.

NEWY and CHIC production updates

2006-11-29T21:16:00.000Z

I pushed the IPv4 and IPv6 unicast and multicast configuration out to all the legacy Abilene routers today for all the current and future upgrade routers. That brought IPv6 up to the new network.

Chris Small and Luke Fowler made great strides in getting the two cities up to snuff from a monitoring perspective. The NOC has them in all their tools. We monitor backbone IS-IS adjacencies and BGP neighbor status at a one-minute resolution. The routers are in the router proxy.

This enabled Matt Davy and I each to work on getting peers up on the New York router. Nysernet's BGP IPv4 BGP session is up (the first US connector, yay!), and the NOX underpinnings are complete. These are new connections that are running in concert with their old New York router connections, so we can ensure things look kosher before turning down old circuits.

The Ciena install teams are making good strides in both Chicago and New York. They should be done with their work by Friday.

Rob Vietzke will be back in New York tomorrow to do some testing of the iHDTV equipment for a plenary HD event. While he's in town, he'll be running some additional patches from the Level3 bulk panel to our Infinera at 111 8th Avenue to help support some other demos. Hopefully those fibers arrive in time.

Our POTS lines in both Chicago and New York are delivered and are ready to be plugged in.

NOX is up/up in New York

2006-11-28T19:35:00.000Z

The NOX temporary wave to New York is up/up in New York. Matt Davy will be working on their interface configuration and they should be passing traffic shortly.

Kansas City and DNS

2006-11-27T23:53:00.000Z

Tom Johnson was in Kansas City today to walkthrough the suite. Looked good except for a few missing details. We'll have to follow up with Level3 on each of those.

Rob Vietzke's video work in New York went off without the HDTV encoders, so he'll need to return on Thursday. He pulled back some Ethernet cable for the Ciena installers to use and noticed that Verizon had delivered our phone line. Apparently it went to the wrong rack and he didn't have an RJ-11 coupler with him, so we'll get that plugged in on Thursday.

Caren got colo worksheets into Level3 for Raleigh, Nashville, Louisville, Indianapolis and Atlanta. We're going to be in a bit of a crunch on those.

We figured out the DNS question. Looks like we're going with net.internet2.edu. The new NOC account will be noc@net.internet2.edu. I2 is sub-delegating that zone to us so we can auto-generate our DNS off our database. Luke Fowler is already starting to work that up.

I worked on the temporary NOX circuit over the Level3 transport network between Boston and New York. At the end of the day, we were able to get light on the New York side, but not on the Boston side. Brian Mollo from NOX confirmed at their rack that there's a missing jumper from Level3. We'll try and get that wrapped up tomorrow.

We're having some very odd problems on the HOPI Chicago management racklan. Devices keep disappearing from the network, but only from certain points of view. Like, we can ping a device from IU, but not from the racklan itself. We can ping devices from some racklan devices, but not from others. They're all in the same subnet and the arp tables look good. It's almost like there's some hidden layer2 ghost in the machine. I mention it because our Chicago Abilene interconnection runs through the HOPI Force10- which is currently unmanageable. Andrew and I looked at it this afternoon and can't come up with a reasonable idea where the problem is. We clear the arp table on the switch and some things come back up...sometimes. Sometimes they don't. And they don't come back up from all points in the world either. Some devices just reachable on the racklan, some just reachable from off the racklan. Clear the table again, and the device reachability reverses. VERY strange. Need to look at it in the morning with a clear set of eyes.

Roadmap for the next week....

2006-11-25T17:05:00.000Z

One week left before the I2MM and lots to do. A preview of the hot-button items this week:

NOX 10GigE to New York

Level3 delivered it on Wednesday after everyone had left for the holiday. We're good to go in New York. I have it running to a loop from 32AofA to 111 8th Avenue. No light from L3 yet, but I need to call it in first. I'll do that on Monday.

I2MM HDTV Demo Testing
Rob Vietzke and a small crew will be in New York on Monday to get the logistics worked out for a video conference to take place during the I2MM plenary over the new network. Gotta make sure that moves smoothly

Ciena Installs
Ciena will be doing the CoreDirector CD install and turnup in Chicago and the Test and Turnup of the chassis we installed in New York last week. Not sure if we'll have the OC-192s up between them before the I2MM, but there's currently nothing slated for them...yet.

Chicago 710NLSD Install Trip Planning
Since we'll be in Chicago for the I2MM the week of December 4th, we'll want to finish up the metro ring install, assuming the 710NLSD site is ready. Need to work with Jake Sallee on that to make sure we're in good shape for a Monday or Tuesday install. Hell, if the power can be ready by Friday, we'll try and go up early to get it up before the I2MM.

Colocation Worksheets
We've gotta work like gangbusters to get the colo worksheets done for the rest of the sites on Ring 2. Caren's got Indianapolis in and we need to try and get as many as possible in by COB Friday.

New York and Chicago router monitoring
The only thing holding me back from migrating circuits in New York is the monitoring. Hope to get that all worked out by Tuesday or Wednesday. By I2MM time, we should have several networks passing traffic on the new node!

Hope everyone had a restful Turkey day. Back to work on Monday!

A gaggle of connections in New York

2006-11-21T22:40:00.000Z

Matt Zekauskus was gracious enough to provide some remote hands support in New York for us today.He took care of a few housecleaning tasks and installed some hardware and jumpers to support the following new connections:

Nysernet - the first connector on the upgraded network
NOX - should be up tomorrow on a temporary wave from Level3
MANLAN - not production, but ready to be used to transition MANLAN networks shortly

He also fixed the missing default route on the OOB router at 111 8th Avenue, so we have console access at that location.

We still need to do a return trip to get a few items taken care of. The power inverter for the Ciena management servers isn't up ready yet, and all the AC power cables need to be cut to length and re-terminated. There's a whole bunch of documentation that we decided to handle remotely, but there will likely be a few copper cables that will need to be traced by hand.

All in all, things are shaping up nicely and we'll be more than ready to go for the I2 Member meeting on December 4th.

First backbone link is UP!

2006-11-20T21:30:00.000Z

We just hit a major milestone. From the Chicago router:

chrobb@CHIC-re0> show interfaces so-0/2/0
Physical interface: so-0/2/0, Enabled, Physical link is Up
Interface index: 147, SNMP ifIndex: 45

Received path trace: NEWY-re0 so-0/0/0
4e 45 57 59 2d 72 65 30 20 73 6f 2d 30 2f 30 2f NEWY-re0 so-0/0/
30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0...............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 0d 0a ................

Infinera OSC data plane

2006-11-19T22:03:00.000Z

Infinera confirmed that the auxiliary port on the front of the chassis is indeed a routed connection and not a Layer2 passthrough. That explains why I can't ping through to the 111 8th Avenue racklan from the 32 AofA racklan. I'm going to need to re-address the machines in that lan into a distinct subnet. No big deal, but I wish we'd had time to test this before we deployed.

Of course, this would be completely trivial if I hadn't spaced the default route on the console server at 111 8th Ave. Tom was doing the final clean-up work for us when the Level3 techs delivered the 2MB ethernet high-speed IP drop from their commodity network. (We're using those in the router cities as a high-speed backup) He asked me if I wanted a default route toward Level3 and I just wasn't thinking at the time and said "no". Realized after we got back, that it's needed since there's no BGP with L3 over that link. It's on the list of things to fix when we head back. *sigh* It'll take 2 seconds. :-)

New York Day 4

2006-11-17T22:10:00.000Z

Writing from La Guardia....

We started the day out on a down-note with Aaron feeling a bit under the weather. So, we started out one man short. Infinera called me last night about the OSC channel and we agreed to start troubleshooting around 10:30 AM. That meant we had to get the Abilene interconnect to the loaner T640 chassis up by then. Worked on the config last night and this morning.

In order to connect the new network with the old network, we're re-using a pair of Internet2 owned fiber from the 24th floor (new node) to the 14th floor (Abilene node). In order to do that, we have to move an existing Abilene connection to the new router to free up that inter-floor fiber. We chose the OC-192 interconnect between the MANLAN Nortel HDXc and the Abilene T640. That circuit has an OC-48 carved out of it for Surfnet that doesn't currently carry any traffic. Lowest impact. Lowest problem.

We decided to frame the inter-floor fiber as a 10GigE so we had enough OC-192 interfaces to the new Level3 backbone circuit. Tom played remote hands down at the Qwest POP to replace the PIC in the Abilene router. Couldn't get it to come up. Tried reseating and resetting the PIC. Nothing. Good light, just no link. Eventually, I moved the circuit to a different 10G interface and it greened right up. Bad XENPAK. Well, not bad, just not a Juniper XENPAK. Came from another vendor.......

Once we had that interface up, we tried getting the Surfnet circuit to come back up. No amount of config would make it happen. Eventually we saw the SONET circuit throwing RDI-P (Remote Defect Path) errors, which would indicate a problem on the Amsterdam side of the circuit. Talked with Caroline to see if she could follow up with them. Ball's out of our court. Moving along.

And move along we did. 20 minutes to get the router up to production speed and make sure I could route through it to the rest of the R&E world. DONE! Infinera calling!

We did a bit of work on the Infinera to verify that I configured everything correctly. Of course (*grin*) I did and all was well. We didn't get the IP connectivity via the OSC channel layer2 payload working, which means that I can't see the racklan at 111 N. Canal from 32 AofA. Since we could manage both sides, I decided to move on and work on the IP connectivity from Indiana on Monday. We configured an OC-192 up to a loop at the 32AofA side and I was happy. Done with that.

By then, Aaron started feeling better. Tom had been working on Ciena power and just finished. So we moved on to housecleaning items. Tom went over to 111 8th Ave. to plug a client fiber in for the new backbone circuit and run a copper cross-connect to the Level3 high speed IP drop that L3 was installing into our MFP. He got that interface up. We got everything documented and all is well.

We'll likely go back in the near future to clean up the racks a bit. We didn't get the Ciena control PC inverter going and we didn't get a chance to cut all the power cables and ethernet cables to length.

A VERY big shout out to Jim Shaffer for all his work. He did a great job staying on the contractors about the power install issues and making sure we had what we needed. I'm sorry to say we were a lug or two short, and he helped us out. Good support at the POP can make an install go so much more smoothly. I'd hate to think how this would have gone without him around.

Photos are updated. We're going to enjoy a nice quiet weekend before we get to deal with this again next week. :-)