Tuesday, December 23, 2008

"I'm sure it's nothing..just a random nagios timeout error. I'll be right back"

I said that to my wife 3 1/2 hours ago as I climbed out of bed to troubleshoot what I thought was a temporary network latency issue. Hah.

I've spent hours on the phone with Juniper support tonight trying to nail down why my netscreen SSG5 firewalls randomly attack the other cluster members. Tonight was the worst so far. Not only did they fight over cluster master, ns1 actually cratered. It refuses to talk on the network. It won't even see ns2 in the NSRP cluster, even though there's a direct cable connection.

Juniper is sending me the latest firmware, and the colo is going to ship me my firewall so I can re-image it and try again.

Of course, all of that is going to happen at some point after I wake up.