feren: I AM THE MAN (koz)
[personal profile] feren
A sort of skeleton timeline is best for this:
  • It took me two hours to drive into work today
  • Today is Tuesday, so I had that damnable weekly staff meeting this morning
  • One hour into the staff meeting, the entire network here at the home office goes away
  • One hour after that, our troubleshooting has produced nothing, so we pull every single uplink GBIC in our 6509 core fiber router
  • This fixes the problem... but....
  • This leaves us with the problem every floor in the tower we call "home" is now isolated from the rest of the LAN, and all the servers are isolated from the users
  • Plugging the GBICs back in one at a time does not isolate the problem. Instead, the problem just magics itself back into existance after they've all been reactivated
  • One hour after the failed attempt to isolate the piece of equipment that's to blame for this, we decide that the core itself is to blame and reboot it
  • Our problem on the admin VLAN goes away, equally magically
  • Our CEO and our CIO want to know what the problem was, how we fixed it, and how to prevent it from happening again. They find little humor in our answers of "Something, dunno, can't tell ya."

    Bah.
  • This account has disabled anonymous posting.
    If you don't have an account you can create one now.
    HTML doesn't work in the subject.
    More info about formatting

    Profile

    feren: I AM THE MAN (Default)
    feren

    April 2020

    S M T W T F S
       1234
    567891011
    1213 1415161718
    19202122232425
    2627282930  

    Most Popular Tags

    Style Credit

    Expand Cut Tags

    No cut tags
    Page generated Jan. 16th, 2026 10:14 am
    Powered by Dreamwidth Studios