Merry Christmas!

Hopefully you got to sleep in a bit later than I did for Christmas! I got a call this morning shortly after 6:00 a.m. because of a “problem with Axway”. With the exception of needing to have somebody physically at the office log me into my computer, and my failure to write down the production down bridge number correctly, it was a fairly easy outage - Bounce Integrator services to get back up and running.

Obviously that’s easier said thn done. First the cluster has to timeout when trying to kill the processes, then we have to manually kill the processes that hung as a result of bringing integrator down. Once we’re back up it’s usually a matter of finding all erroneous log entries and resending the files from the syncpoints. By some miracle (Christmas?), the majority of the files that were hung when we killed integrator actually finished processing by themselves when we brought it back up. As such only 4 files needed to be manually re-sent! Of the files that needed sending again, only one erred out this time, and it was only because a temp file was created upon initial attempt to encrypt the file. The fix for that is simply to delete the temp file that was created and re-send.

All in all we actually suffered about 67 minutes of outage, 42 of which happened before I got a call. Unfortunately 25 minutes recovery time is not good enough. We were able to diagnose the problem instantly (though still no root cause to outages occuring en masse since September), but my not having the bridge# (saved to cellphone), inability to log in (need to get a laptop), and our having to wait for the cluster to timeout (manually handle integrator shutdown and disable failover on cluster next time) really slowed things down.

Fortunately though, barring another outage, this should keep us within our 99.7% uptime requirement for December. For January we increase to a 99.8% uptime requirement though, which means we need to come up with root causes and fixes for these issues ASAP. I know what I’ll be working on tomorrow!

Merry Christmas, Happy Holidays, Season’s Greetings, etc.

Tony