Friday, September 14, 2007

When one problem exposes another

Sometimes the network Gods have a funny way of toying with your mind. We noticed some L3 incompletes on the new CalREN circuit yesterday, and we pointed the finger at the optical levels on the CalREN interface. I worked with Level3 early this morning to clean the fiber path between I2 and CalREN. They found a bad port in the path at 600 Wilshire that was causing us to launch -14.3dB toward CalREN- just a hair under the tolerance specs of their card. After we cleaned that up, the L3 incompletes persisted. Turns out it was BPDUs that hadn't been filtered all along. Had they been filtered, we may not have caught the light level problem.

Someone was trying to tell us something. :-)

Anyway, CalREN is clean as a whistle now and should be shifting traffic tonight. I'm going to bed.