Outage Today

[Geoff Winnington-Ball (RIP)]

Sorry for the hassle. There was a mailman process on the server which went out of control early this morning and sucked up all the server's memory. Their monitoring equipment didn't pick it up and it was only until I got SUNRAY MINOR to get in through SSH and diagnose the runaway python script (he's a unix networking dweeb ) that we were able to get them to reboot... shouldn't happen again!

Geoff

[Geoff Winnington-Ball (RIP)]

Unfortunately, it seems they were still ironing out the bugs yesterday afternoon and evening. Hopefully it's done now.

What happened was that they changed hostnames for the server, which screwed up pop mail for some users. I wasn't hit too badly - only a few 'bounces' I heard about - but some lost mail service altogether.

They then frantically wrote and executed a script to resolve the hostname redirects on incoming mail, and that was what screwed up. With all server resources used up by this runaway script, http requests were being bounced, with you out there being given "Server Hangup" or similar messages.

The effect of the runaway script is about the same as what you experience when some program (usually Microsoft) suddenly locks up your machine, and all you can do is do a power-off reboot (losing all your work, of course).

Again, sorry about that... for us, here, timing couldn't have been worse...