Let me put it this way...
Sep. 25th, 2001 03:34 pmA sort of skeleton timeline is best for this:
It took me two hours to drive into work today
Today is Tuesday, so I had that damnable weekly staff meeting this morning
One hour into the staff meeting, the entire network here at the home office goes away
One hour after that, our troubleshooting has produced nothing, so we pull every single uplink GBIC in our 6509 core fiber router
This fixes the problem... but....
This leaves us with the problem every floor in the tower we call "home" is now isolated from the rest of the LAN, and all the servers are isolated from the users
Plugging the GBICs back in one at a time does not isolate the problem. Instead, the problem just magics itself back into existance after they've all been reactivated
One hour after the failed attempt to isolate the piece of equipment that's to blame for this, we decide that the core itself is to blame and reboot it
Our problem on the admin VLAN goes away, equally magically
Our CEO and our CIO want to know what the problem was, how we fixed it, and how to prevent it from happening again. They find little humor in our answers of "Something, dunno, can't tell ya."
Bah.
Bah.