Shit!

[feren]

In a fit of pique (and stupidity) with a looping process I did a pkill -9 ovw command on our management console, which instructs Solaris to send a SIGKILL to every process that has the string "ovw" contained in its name. Unfortunately I forgot that ovwdb has the string "ovw" contained in it, so I completely destroyed that process along with the process that had hung on me. Once ovwdb went away netmon crapped out, because it cannot run without constant communication with the data warehousing that ovwdb provides.

So not only did I shoot myself in the foot, I shot myself in the foot in a spectacular fashion. My document writing must now be put on hold while I try to recover the processes and get the tables normalized again.

Bravo, Feren. Bra-fucking-vo.

Comments

[tuftears.livejournal.com]

Ouch! Sorry to hear that. Hope you have backups and the like! Ops has pulled some boners here as well, but we've dealt with them pretty well.

[kinkyturtle.livejournal.com]

That reminds me of the time that spot found a lot of spurious files created on her computer for some reason, and they all had ASCII gibberish for filenames. She happened to notice that most of them had a question mark in their names, so the first thing she instructed her computer to do was delete all files matching the wildcard *?*...

Yeah. WHOOPS.

[feren.livejournal.com]

On a completely unrelated note, it appears in your icon as if there is a small star going super-nova over your left shoulder.

[kinkyturtle.livejournal.com]

That's a ball of energy being shot out of my eyes, to destroy a nearby Pikachu (not pictured). Neat, huh?

[feren.livejournal.com]

You destroyed Pikachu using only your eyes?

You magnificent bastard!

[tetragon.livejournal.com]

Heh, ouch!

No sysadm is immune from this kind of thing. My worst oh-no moment was when I stupidly assumed killall on HPUX might be the same thing as that on free OS's. No, that bought our V2200 db server down to single-user. Fortunatly (in this case) the progress db processes were (FSSR) running as root, and the dbs were not corrupted, thank god.

I did have to explain though, why I stopped the productivity of about 200 workers for almost an hour. :)

[captain18.livejournal.com]

"They looked like they needed a break."

[feren.livejournal.com]

Christ, if only I were that quick-witted in person.

[ronbar.livejournal.com]

I have a mutual non-aggression treaty with openview; un-fucking the maps after a major outage usually involves stopping everything, restarting it all, and waiting a while for it to synchronize. That can easily take thirty minutes, and if lots of things are flapping at once, the whole thing is just useless. It never catches up after trying to change the status of all 10,000+ items at once, even if they all appear (to the collection station) to have gone down and come back up just once.

By luck I've never used pkill, since I tend to use commands that are mostly the same across Solaris and Linux. That way I don't have as many problems with non-existent commands and bad syntax when I'm switching between them. I also found (and use) of the rare older Sun keyboards with a PC-style layout on my workstation. I'd be completely unable to type on either type of keyboard if I used the brain-dead Sun layout.

[feren.livejournal.com]

[. I'd be completely unable to type on either type of keyboard if I used the brain-dead Sun layout.]

Surprisingly enough... I like the standard Sun keyboard layout (with "control" in place of the caps lock, etc). I really have no idea why that is.