New York, POTS and travel

Had a call with the Nysernet folks this morning. Sounds like they're on track for delivery. The racks have been bolted down and about half the overhead infrastructure is finished. They have our DC power controllers, but not our Telect panels. Since they don't have production AC power at the site, we're rack mounting two inverters for our AC-powered equipment. The original thought was to terminate the Marconi-supplied power feed directly into the inverter, but that idea obviously flew out the window when we realized that the inverters don't have any safety breakers on them. So, we're "shimming" a simple breaker-based Telect panel between the BDFB and our inverter. But, of course, they haven't come in yet. "Should be here any day". In the meantime, I'm trying to track down some loaner panels, though that appears to have hit a roadblock. We should still be OK as long as the Telects get here before we do the install November 14th. It may be another visit for the electrician, but at least it will get done. (32AofA is a union building, so we can't touch the house power)

The POTS line miracle was somewhat dulled when we heard AT&T can't provide the DLRs. I'm working with Level3 to find out if they need a formal letter for those, or if we can just get the termination location at the building MPOE and order the cross-connect without something on AT&T letterhead. I knew this would be a stumbling block. Tom Johnson is all over this right now.

Got our travel plans worked out for Chicago and New York. It's set in stone now, baby. There's no turning back!

A possible miracle in the works...

Tom told me right before I left the office on Friday that the Chicago POTS line is scheduled to be installed on Thursday! We're supposed to have the DLR on Monday, which hopefully will give Level3 time to run the cross-connect from our suite to the building MPOE. What a wonderful bit of news as we head into the weekend.

Commodity Peering Service to the rescue...

As a completely separate topic, Abilene started turning up it's content provider peering this week. But that's not what I want to talk about. I'd like to focus on how this will help us "glue" the new network to the old network in Chicago. Internet2 purchased metro fiber between Qwest (current Abilene node), 710 North Lakeshore Drive, and 350 N. Cermak, where the commodity peerings physically interconnect at Equinix. This fiber runs via 600 W. Chicago St, the same location as the new Internet2 node in Chicago.

So, the current thinking is to "shim" the new Internet2 router in between 350 Cermak and 710 NLSD. Easier said than done, I'm afraid. The Looking Glass fiber that I2 purchased runs through the 3rd floor of 600 W. Chicago. The new Internet2 node is on the first floor. See where this is going?

There's no fiber between the first and third floors...right now. Level3 is working on a solution. We've already told them that if they can make this happen, we owe them some beer. I'll make that a case of beer if they can make it happen in time for our Chicago install. Getting fiber blown between floors can be tricky, so this will be especially impressive if they can pull it off. Go team!

Friday loose ends...

Got some loose ends tied up today:

Jay found some 23" rack ears from our power controller vendor. We overlooked the 19" controller in the 23" Infinera racks in Chicago and New York. As a backup, we could have used some 1RU extenders that we have in stock, but it would have been kludgy.

The multifunction panel we're ordering from Level3 in Chicago will be installed in time. This enables us to terminate our fiber to the Looking Glass metro fiber that we're going to "shim" the router into. (see my previous posting)

We got our badge requests and travel arranged for the Chicago trip. New York is nailed down to us arriving the morning of November 14th. While the rest of the networking world is in sunny Tampa for Supercomputing, we'll be in a POP in New York. How did I get the short end of the stick?!?

Got the Nysernet badge agreements for access to the suite at 32AofA. Shooting them out to the ESNET folks on the list and everyone else that needs them.

Tom is making progress on the POTS lines orders. It sounds like we can process those without a PO, making life much easier. We still need DLRs for the cross-connects to our suites. I really don't have much faith that we can get this done in time for the install, so I'm abandoning that goal. We can't transition anyone in any city without OOB, so it's at the top of my worry list.

We have a LOT to get done on Monday to make sure we get the orders in on time. I owe Jay Duncan a massive list of intra-rack fiber and copper connections so he can figure out the lengths and the type of fiber to order. Shouldn't be a problem as long as we get it in on Monday. :-)

Early October Broomfield workshop

Greg Wood sent me a photo of the Broomfield workshop in early October. This was the last big meeting between I2, IU and Level3. A very cool conference room that has curtains off to the side that open up to their NOC. I'm told that there's an audible alert on the other side of the NOC glass that warns them us customers are watching. L(3) has a pretty cool NOC setup. Viewing it always leads to the inevitable discussion about everyone's favorite NOC.

Ethernet cables

Jay put together an initial copper cable run list. It's 39 cables at the router sites, each of varying length. Network-wide, we're somewhere around 600 patches. Terminating ethernet cables on site can be a real pain. At that scale, it can take a significant chunk of the day. So, we're going to estimate the lengths ahead of time and get them pre-made. We'll send some additional cable along in case our lengths are off, but this seems the best way to avoid unhappy engineers who haven't made cables in years. Happy engineers are productive engineers.

Nysernet in Buffalo

It sounds like we're going to have a solution in place for the Nysernet folks to transition off the existing Chicago Abilene router. The problem is one of sequencing. Level3 won't be ready with the New York-Cleveland-via-Buffalo route until March '07. Since we're re-using existing Abilene routers in the new network, we need to transition people off them and move them to a new city in order to facilitate the next turnup. Since we want to get Chicago to Atlanta before Level3 delivers the Atlanta route, we need to get all peers transitioned off that router by February. There lies the problem. Nysernet comes into Chicago and the transport won't be there in time. So, they're being put on a temporary wave with Level3 to get them to Chicago. This goes on the Level3 Infinera transport gear, not the I2 Infinera transport. Rob's been working hard at getting some metro fiber in place for them and the pieces all sound like they're falling together nicely. One less connector in Chicago to worry about transitioning....

DC suite

Worked remotely most of the day when I wasn't talking at the NSF. Spent most of the day working up a spreadsheet that shows all our projected wavelength demands. This is purely to meet an early deadline from Level3 for capacity planning. They'll take our demands and put them into a system that will track our capacity and provide us with the metrics we need to decide when to deploy more Infinera interfaces. Eventually this will end up in an IU-managed circuit database. We already have a weather map ready to go. The data just needs to be populated. I expect we can do that within the next few weeks.

Visited the McLean, VA suite for an acceptance walkthrough. Looked good for the most part. The bulk cable to the MAX suite on the other side of the colo area wasn't in yet, and our multifunction panel (MFP) wasn't racked up. We were fortunate to catch Level3 in the act of testing our bulk fiber to their transport area. You'll see that setup in the picture I've attached. A bit of explanation...

The Level3 transport for the new Internet2 Network, via Infinera DTCs, is not located in Internet2 suites. All that gear sits in the Level3 transport area in a totally separate part of the POP that we don't have access to. In order to rapidly provision interconnects from the I2 suite and the Internet2 Network transport gear, we have provisioned bulk cable to all of our suites. The numbers vary, but in general, it's 96 strands (46 pair) of fiber in each direction at the router nodes, and 48 strands (24 pair) in each direction at the grooming sites.

So, it was good to catch the tech today so he could tell me how the ADC panels are numbered. Looks like they ship with the ports already labeled. It's different than I was expecting, and the timing is good here since I'm going to work on fiber panel tracking tomorrow. Now I know how that FDP is labeled. Need to work on the other three vendors. :-)

Fiber tracking and grooming announcement

Writing from the Indy airport on my way to DC. It was a frantic morning trying to get things tied off from Sunday before I left. Got Chad Kotil looking at the old Abilene NGAR transition pages from 2003. He's using a web system called WebGUI that should allow multiple users to edit the pages rather than have me maintain all of them like last time. That'll be important for my pending hiatus around March.

Talked with Ed about fiber panel tracking. He hadn't gotten around to modeling the panels in the db yet. We have four different FDP vendors and models right now, so we need to make sure we keep the numbering system and port reservations clear. I'm working on this on the plane.

Rick Summerhill sent out the official word on the grooming box vendor. It's the Ciena CoreDirector series. We'll be doing a full CoreDirector CD in Chicago (complete rack solution) and the half-rack CoreDirector CI in the other sites. Right now, I'm trying to locate someone in the community that can give us a few days worth of remote access to a lab or non-production CoreDirector. Our monitoring group wants to get a few days to write tools in advance of their deployment.

Got dark fiber assignments from Level3 on the non-Level3 ends of the metro circuits in Chicago and New York. The info is missing the rack numbers, so once I have those, we can get the final patches within the 32 AofA Fiber-Meet-Me-Room (FMMR) in New York and 710 North Lakeshore Drive in Chicago.

Rob's making progress on the NOX temporary wave over Level3's own Infinera network to New York. That's being provided so we can transition the New York router away before the New York-Cleveland route via Buffalo is delivered in early March.

Had some discussion with Jay and Jon-Paul on our warehousing and inventory vendor in Indianapols, Telamon. At issue is the way they bill for received items. When we handle receipt and shipment of equipment, we take all the small items (power cords, optics, cards, fuses, etc.) and throw them into the box for shipment. Telamon charges by the box on the outbound and we didn't set up any stuffing arrangement (it was cost prohibitive). They may be able to do simple stuffing and putting loose items in the outbound box at a low enough price, or we may just go pick all that gear up and ship it out as part of the install kit. Discussions ongoing...

Got 9 new HP GigE SX optics ordered for a second router interconnect to the HP switches at the router nodes. We'll use that when the 10G interface is unavailable (during testing or control plane work)

Sunday, bloody Sunday

Who says work stops on the weekend. It just means the phone stops ringing!Got a lot of conversations going today.

We need to decide on a circuit naming scheme very shortly so Level3 has it for the temporary waves we're getting to help with the transition. Right now, the thinking is "I2-Acity-Zcity-customer-index. The city names will be the first four or five characters from the city rather than the CLLI code. Much easier to read. Need to figure out how to handle New York, Seattle, and Chicago where we have two locations.

Nathan Lucas put together a one-page draft of the access procedures for customers that are co-locating in the I2 suites. Looks good, though I had a few suggestions. When it's final, I'll post.

Getting the ball rolling on a status webpage. We did one for the Abilene upgrade in 2003 that worked well. Had some good metrics and snapshots of the transition. It's not something we've spent enough time thinking about. We may still end up using the wiki, but the consensus seems to be that the wiki may not be well-suited, given it's lack of structure. Again, we need to think about it some more.

Gotta get the ball rolling on a more detailed discussion about the colocation at 710 North Lakeshore Drive in Chicago. Tim Ward, Caren and I need to sit down and sanity check their understanding of our requests. Hope to get that in before I leave or have it happen without me while I'm gone Tuesday and Wednesday.

We're just starting to tackle the problem of badge access to the Level3 sites. This is somewhat tied up in the discussion about colo access that Nathan's already been working on. The short of it is that we're planning to issue badges to all of our customers, though they expire and there's a limit to how many Level3 sites you can have on your badge. That's more of an internal issue between us ESNET since we're the only ones on the nation-wide footprint. Hopefully we can have some sort of quick database app down the road, but it's way down on the development priority list. Tracking fibers is so much more important!

Deployment sync

Big meeting via video last Friday with most of the key I2 and IU players. We all have our marching orders. Level3 brought up some changes to the architecture of the management network access, our hook into the Level3 Infinera transport gear. There's a call on Monday to discuss that in more detail.

Level3 is dropping I2 and ESNET DS-3 circuits into Indy and Berkeley for our direct monitoring access. They're private circuits taken back to Atlanta and Denver, respectively. We decided that we can procure metro dark fiber to the Level3 POP in Indy rather than puchase off-net services from Level3 to get to the IU campus. So, that means a DS-3 router in the Level3 POP with a GigE back to campus over some fiber. The challenge there is going to be obtaining the dark fiber and getting the Level3 POP ready in time for the first route delivery on November 17th. The changes Level3 proposed on Friday may also push us out a few weeks past the circuit delivery. If we're a bit late with the management circuit, it won't hurt us, but it would sure be nice to keep those two in step. I'll see how creative we can get on Monday morning.

Gotta get rack elevation diagrams to Nysernet in the AM so their electricians can mount and terminate our Power Distribution Units (PDUs). We use Servertech rackmount DC and AC power controllers for any devices we think we'll need to remotely power cycle. They have ethernet and console connections that can be accessed to remotely manage the power. We typically don't like to have our engineers terminating house power into our controllers. It can be done, but it's riskier (since we don't control the breakers) and the cable tends to be more difficult to work with because it's much thicker. Cutting, lugging, lacing and terminating #2AWG power cables can be a 2-3 hour affair, sometimes.

Looks like we have some tentative travel plans worked out for the install. I'm nervous about New York, trying to find a hotel room near Thanksgiving. Nysernet has told us that it can be nearly impossible. Depending on our weekly status call with them on Tuesday, we can get our tickets ordered.

Right now, the current thinking is:

• Chicago Install #1 (router, grooming, PCs) - November 6th


• New York (Infinera metro, router, grooming, PCs) - November 13th

Those in the community will recognize the week of November 13th as Supercomputing 2006 week. I'd love to avoid it, but if we wait, we're looking at Thanksgiving or the week before the Internet2 Members Meeting. Since there's an Infinera turnup in New York, I'd rather have more time than less time since it's new gear for us.

I'm off to Washington D.C. on Monday afternoon for a meeting with the NSF. I'll be walking through the McLean POP and hopefully accepting it. Onward!

Timing is everything

Got the group together for Infinera training today. Started out early in Bloomington. A bit of a mad scramble before Mike Reilly got here to find 10G client optics that would work with the Infinera in Bloomington. Had to borrow an LR X2 module from the campus guys. Did some configuration in the Wrubel machine room and walked through the cabling. VERY simple to cable up. Basically just a line in/out and a loop up to the DCU. Nice and clean, no fuss.

We did run into a problem in Bloomington after we drove to Indy. Our fiber guys had left the jumpers dangling at the patch panel when they did their testing. Had one of the campus engineers go over and plug them back in and we were in business.

Along the way we learned that the Infinera dosn't provide clocking. It's essentially a transparent optical system that does 2R regen. It would be as if we plunked our Chicago T640 and our New York T640 right next to each other and hooked them up via a 3 meter fiber. Right now we derive SONET clocking from the Qwest long-haul network. There's a bit of disagreement internally over whether we need to worry about getting BITS timing to our Junipers. I'm not convinced its needed, but I need to do some digging before I start getting too emphatic on this point.

The GMPLS stuff on the Infinera looked good. Overall it was very easy to create circuits and monitor their path. It sounds like their EMS platform has some very good views into the network that will enable some rapid problem identification. I'm not sure how much we'll take advantage of that on Day 1 given the very simple metro rings in Chicago, Seattle and New York, but it's good experience on the platform as we move forward with the Level3 relationship.

It sounds like Rob found a few niggling items in the New York suite, but nothing that can't be overcome. I've got a few messages in my inbox from Level3 about circuit naming conventions that I need to bring up tomorrow. We're having a four hour video conference with representation from all corners of the project to discuss the gritty details. I've been working this evening on card projections and getting some diagrams updated. There's a lot of material to share.

As for communication, it sounds like I2 is game to let this blog keep rolling. The family may increase as they may start posting items, too. At least, that's one of the plans. They have a wiki-based blog that contains more formal communiques about the network status. I can't blame them for not wanting to mix in my ramblings about shipping logistics. :-)

An Infinera lights up...

A busy day today with most of us scattered about doing different things. Both lab Infineras are powered up with ethernet and fibers dropped to both systems. We got a bit gutsy with the heat-shrink, trying to shrink it with a lighter. Once I started working toward the top of the rack, I realized I was right under the smoke detector in our machine room. Normally we have a paint stripper, which does a great job of generating heat. But today, that was in Indy. So, I went home and grabbed my wife's hair dryer. Infinera is scheduled to arrive tomorrow morning to begin training at 9AM. We're looking forward to it.

After a bit of back and forth about the out-of-band network and it's design, we finally got a quote for the Cisco 2611 routers we'll use to get asynch access to our racks. Matrix Integration tells me that they'll have them to us by next week. That's really good news since I was looking at the schedule last night and saw that the projected 25 day lead time pushed New York and Chicago out to late November if we wanted OOB at install (which we do). Now we need to get the POTS lines ordered. That is going to be the most likely holdup, but it is what it is.

We have a good line on some ADC fiber panels to use for customer terminations. There's a meeting on Friday with all the I2 and IU folks involved in the transition to do a deep dive into every aspect of the network. I need to provide a ton of documentation. Most of it I already have, it just needs some updating and a few formatting changes to make it more readable.

Pittsburgh and Philadelphia were ordered today, though there's some question about whether or not we have enough rack space in Pittsburgh. We may need to bleed into a second rack, which is extremely undesirable.

Rob Vietzke is visiting New York for us tomorrow to do a suite acceptance. I'm going to do DC on Tuesday next week, when I'm in town.

"The colors are different in Europe..."

Matt is in Chicago doing an I2-related HP switch install. He's still there, but he caught me on IM and relayed his day so far. Apparently the AC power plug was provisioned as a NEMA L5-20P. That's a "lockable" plug that has special notches on the male end of the plug so it can be twisted in the receptacle. This keeps it from falling out, but you can't use a non-lockable plug in the receptacle. Andrew found an Ace Hardware nearby and Matt got the lost L5-20P plug. When he stripped the wire on the HP switch, he found brown, blue and yellow/green wires. Apparently it's a European standard. Once that was figured out, he found the plug and fiber trays to be too high to reach without a step-ladder, which he didn't have. Makes me happy that we caught the fiber tray height problem at the new I2 POP in Chicago.

Apparently others in the POP have lawn chairs in their suites for days like today. That's an excellent idea. We often end up sitting around waiting for technicians to call back, the NOC to get a ticket open or someone to configure a device. Why not do it with a footrest? $10 at Target. I'm sold.

Grooming install logistics

Had a call with our grooming vendor today to discuss shipping and install logistics. Given the number of international and domestic connections in Chicago, that site is somewhat of an exception. It's getting a full-rack grooming box instead of the more standard half-rack solution. The chassis ships in the rack from the vendor. They tell us it's pretty heavy. Sure enough, the docs say 950lbs fully loaded. We're not going to fully-load it, but let's be honest: anything over 50lbs is enough to wear a pasty-faced engineer out in 1.7 seconds. Us cornfed folk at Indiana University aren't complete weaklings, but we're not sneezing at 950lbs. That's a lot of corn! So, we may get the vendor or Level3 to do the install. Decisions. Decisions.

Oh, and we discovered today that we're short one DC power feed on the grooming box. Normally we have a fuse panel or power controller shimmed between house power and our equipment, but Chicago is an exception. House wires directly into the PDU, which means we need one more redundant circuit. Still working out the details on how much power each of those need. I still have a bad taste in my mouth from when we ordered 140A of -48VDC for the Junipers on Abilene and ended up using 20-30A, real-world. We shall see.

Loaner T640

Looks like the loaner T640 from Juniper arrived in New York yesterday. Jim Shaffer (Nysernet) is working with remote hands to get it into the suite. Apparently the crate won't make it through the doorway. :-)

Monday morning catchup

Lots going on today. First off, an apology for not posting more about the Chicago trip. It went off and I had a long entry that I worked up in the car while driving back (well, Nathan was driving). When I synched up with our blogger account using my cell phone, the entry was overwritten. Clearly, there's lesson there.

The short of it is that the suite visit went fine, aside from a few niggling items. There was an unfortunate miscommunication on the time, so we ended up sitting around in a coffee shop for a few hours waiting for the Level3 escort. The 600 W. Chicago building requires a second badge to get in that's completely separate from the Level3 badge, so we'll need to worry about getting those. As the Level3 tech said, they tried integrating them, but one system was "32-bit" and the other was "64-bit". ;-)

Today was spent pushing e-mails to make sure we plant the seeds for some future discussions. We got the HP switch PO out the door, which will make things tight on our management and observatory networks. I'm waiting on a quote for the Cisco routers that we'll terminate our POTS lines in for the out-of-band management network. Oh, and we have to order all the POTS lines. This stuff is seriously keeping me up at nights, so I have a vested interest in getting it done sooner rather than later.

The Infinera grounding screws showed up, so we can now terminate the power at both IUPUI and Bloomington. Of course, the lug crimpers are in Indy, so we need to wait for a mule (Matt) to bring them back. Infinera is coming on Thursday. They're in luck since UITS is throwing it's first ever Chili Fest on the same day. We're thinking of telling the tech that it's all in his honor.

Let's talk patch panels. Seriously, these things are the other reason why I'm not sleeping. We need them for all the connector LOAs. MAGPI and a few others are blocking on fiber orders until we can give them an LOA that specifies the tie-down info within our suite. Caren's got a line on a module ADC panel. We're going to go with 14 pairs of SC-connector fiber at the grooming nodes and up to 24 pairs of fiber at the router sites. Thankfully, the specs show the port positions so we can say with some level of confidence that we know where the fibers will land on the panel.

Now we have to worry about tracking all this. In the old days, we would have done it with a spreadsheet. But this is the 00's, baby. There are wonderful inventions called databases. You may have heard of them. Fortunately, we also have the requisite database programmer that's already been thinking five steps ahead of yours truly. Big props to Ed Balas for the foresight to get this done. Now we just need to hammer out the details. Hoping to do that tomorrow.

We started thinking about the logistics of the management network for the grooming system. OK, we thought about it before today, but now we're doing the "how do we get fiber between this building and that building" dance. Think we have some good solutions for I2.

Picked up the HD video camera for the installs. Yes, it's a bit of a toy, but will be useful. I had fun making the rest of the staff sick by sticking them in front of our 65" video display and making them watch video from my perspective of a walk down the hall. Gotta have fun at least five minutes out of the day.

Away to Chicago we go

Chris Small, Nathan Lucas and I are heading to Chicago this morning to check out our first Level3-delivered suite for acceptance. I've got Caren's hard-won diagrams from Level3, a measuring tape, and a camera (no video yet, unfortunately). We'll be off the air for a while until we get back sometime late tonight. 5 1/2 hours up, 5 1/2 hours down. What a wonderful way to spend the day.
-Chris

New York power and shipping

So, apparently we *can* bring equipment up the freight elevator at 32AofA, but we definitely need to be at the loading dock when the equipment arrives to receive it. The grooming vendor has said they can have the carrier take it up to the 24th floor, though, so we're good on both fronts. Hopefully we can get that specified for all shipments.

Nysernet has some issue with us wiring their Marconi DC power plant directly into our power inverters, so we're going to change things up a bit and shim a 1RU Telect breaker panel in the middle. Jay and Caren are coordinating to get that to them on time.

And Jay's at Telamon right now to work out the details on getting the rest of the Sentry's shipped to New York. I see that ESNET already beat us to the punch and shipped theirs out today.

The Loan Chassis, and his sidekick, "Simplicity"

Well, our lives got a lot easier today with the news that we will indeed be getting a loaner T640 from Juniper to help out with the transition. Sounds like we'll have it for four months. It removes the need to hot-cut New York and allows us to do the install a bit sooner to be ready for the new circuits. The order is now:

• Chicago - November
  
• New York - November
  
• Washington D.C.
  
• Atlanta
  
• Indianapolis removal (ship to New York)
  
• Kansas City
  
• Houston
  
• Salt Lake City
  
• Denver removal
  
• Seattle
  
• Los Angeles
  
• Sunnyvale removal
  

I think we can do DC pretty quickly after the I2 MM. There's some talk about doing it beforehand, but that means changing the network configuration the week before the member meeting, which is something we'd like to avoid. It doesn't get us much in terms of capability and would really force us to move too quickly after the first route is handed over November 17th. Remember that Supercomputing and Thanksgiving bring everything to a complete halt during November!

Argh! Shipping madness!

Shipping to a POP is one of the most insane things sometimes. You need to make sure that you have proper access to the dock and that the dock will accept your shipment. Sometimes you need to meet the equipment. If you meet it, do you have the appropriate permissions to use the freight elevator? Is said freight elevator tall enough for your shipment? Does your carrier take the equipment all the way up to the floor your suite is on? Does it matter? How long can the POP store the equipment. Will it be secure? Who has the responsibility to insure the equipment while it's sitting there? Once you're done with the equipment, what do you do with all the trash?

At least I know all the questions to ask, but there's always one, tiny nagging detail that gets in the way. A ladder rack that's a half-an-inch too low or something along those lines. It's really a full-time job, which is why Nathan is going to tackle it. Godspeed.

-Chris

Lab Infinera installed

Our Infinera gear showed up while most of us were in Broomfield last week. The dock guy was going to kill me if we didn't get it out of the hallway so Andrew, Jake and I racked it up. Looks good, though it don't gots power yet. I'm waiting on some rack extenders and stranded 6AWG cable for the DC power. I'm initially disappointed that the power terminal is on the front of the chassis. Might make it challenging to run power.

Stupid me, forgot that there was a DCU unit in Bloomington. With the grooming loaner coming shortly, there may not be enough room in our hard-won lab rack in the WCC machine room. *sigh* A big props to the campus folks who got the power and fiber runs done for me on short notice. All my nagging paid off!




Hopefully we'll get this bad boy powered up tomorrow. Infinera's coming in next Thursday so we can cable it up. I need to think of names for the boxes. I'm thinking Herman (Bloomington) and Wells (Indy).

-Chris

POTS or ISDN

We've gone back and forth about the best method to provide out-of-band access at the new network. The big router sites are mostly taken care of with some high speed IP drops from Level3. (The exceptions are Chicago, because it's off-net, and Seattle and NYC, where our routers aren't going into L3 space) The rest of the network is left to fend for itself in the event of an outage....

Of course, we wouldn't leave a grooming node or remote router site just sitting by it's lonesome during an outage with nothing to do but give us headaches. We looked at a lot of different ways to provide fail-safe access:

• in-band STS channel with access from both sides of a redundant ring- too risky if the control plane blasts the management channel away
  
• high speed IP drops with IPSEC tunnels to IU - not available in some sites; expensive
  
• DSL - difficult to order; DSLAM hardware and firmware compatibility issues; some carriers won't let you terminate in your DSL modem
  
• ISDN - low bandwidth; difficult to troubleshoot and order
  
• POTS line - low bandwidth
  

Eventually we settled on a hybrid plan whereby we use the in-band STS channel with fiber path redundancy for protection. We'll get POTS lines at all locations as a fall-back. We know that won't give us the bandwidth we need to manage the grooming/DWDM boxes through their GUI interfaces (at least, it would make things abysmally slow), but we can bring the STS management channel back up using canned TL-1 commands that we keep handy in the event of a catastrophic outage. POTS is also easier to troubleshoot than ISDN. Pick up the phone and you either get a dial-tone or you don't. See carrier for any further problems

Now the challenge is going to be getting the last cross-connect between the carrier's punchdown block in the POP to our suite. Level3 wants a DLR from the carrier, which can be challenging to get when talking to your average sales rep. Need to approach that issue quickly. I may hand it off to someone else so it doesn't block on my time.

We initially thought it would be useful to stick a phone in the POP. The Cisco interface card we're looking at using has a second RJ-11 jack on the back for an extension. Caren brought up the excellent point that life would be bad if we had to rely on the POP techs to hang up the phone when they were finished using it during a hands and eyes request. We'll settle with sending our install teams out with a handset they can plug in.

-Chris

While a photo is worth a thousand words....

Yeah, we try and take photos of every install, change, card slot, and POP visit we make, they always suffer at the hands of the person taking the photo. There's also no really good way to capture the overall size and scope of a particular suite or POP. So, I'm working on getting an HD video camera to film the installs. We've done it a bit with our digital video cameras already, but the video is small and jumpy. Luke wants to film the whole install and then post-process the video into time lapse. And I want to shoot a documentary on the install. Yeah, we'll see....

Currently looking at the Sony HDR-HC3.