When one problem exposes another

Sometimes the network Gods have a funny way of toying with your mind. We noticed some L3 incompletes on the new CalREN circuit yesterday, and we pointed the finger at the optical levels on the CalREN interface. I worked with Level3 early this morning to clean the fiber path between I2 and CalREN. They found a bad port in the path at 600 Wilshire that was causing us to launch -14.3dB toward CalREN- just a hair under the tolerance specs of their card. After we cleaned that up, the L3 incompletes persisted. Turns out it was BPDUs that hadn't been filtered all along. Had they been filtered, we may not have caught the light level problem.

Someone was trying to tell us something. :-)

Anyway, CalREN is clean as a whistle now and should be shifting traffic tonight. I'm going to bed.

CalREN almost migrated in Los Angeles

CalREN engineers shifted their traffic over to the new Internet2 link, and noticed a very small trickle of errors on the port. Light levels are showing up right below the threshold, so we'll need to do a bit of cleaning there. Look for that to happen soon. 

NGIX-W/NREN migrated to Salt Lake City

I migrated the Sunnyvale NGIX-W and NASA connection over to the Salt Lake City router tonight. Always a bit tense when you move that many peers, but this went as smooth as silk. A big thanks to our friends at ESNET who provided a concurrent GigE interface to the Ciena in the Sunnyvale POP so that we could backhaul the circuit to Salt Lake City.

HOPI IP migrated in Los Angeles

We were able to get the HOPI 10GigE interface up in Los Angeles today, across our metro fiber between 600 Wilshire and 818 W. 7th. We tried to get the PAIX 10GigE up to a loop in Sunnyvale, but we're having some metro problems in LA. The signal just ain't getting between the buildings. We have a ticket open to troubleshoot that in the AM. 

LEARN migrated UT Austin behind them on Friday night, so they're now shut down on Houston. That leaves three BGP sessions left on Houston. 

I need to schedule CENIC on LA- the last BGP session on the old Los Angeles router. 

After that, it's just PAIX, NGIX-W and Oregon in Sunnyvale. I'll probably move the NGIX/Ames VLANs this Wednesday, along with PAIX if I can get them up tomorrow. Oregon may be a few more days out than that. 

Pacific Wave, CENIC and LEARN and all the rest...

It's been a month or so. Let me see if I can catch everyone up. We've been mainly busying ourselves with getting some fiber patches and circuits lit on the west coast.

I just migrated all of the Pacific Wave Los Angeles peerings over to the new Los Angeles router. Once we got the link up, it was a painless hot-cut.

We did some metro fiber troubleshooting in Los Angeles this evening and the CENIC/HPR connection is up to the new router. We'll need to coordinate a time to swing their peering over to the new circuit.

LEARN a few weeks ago, and the Texas schools are all working on aggregating behind them. Our piece is done.

USF also looks to be up in Atlanta, so we'll probably be able to shut the old M5 down pretty soon.

We're close on Oregon and PAIX. I'm hoping to get all our pieces worked out this week. There are some interfaces being shuffled around in LA to get them from the old router to the new one. Those should be in place by Friday. We very well could have everyone off the old routers by the middle of next week, if things click. Probably more realistic to expect two more weeks, though.