Our plans worked – we fixed the client’s server – and as a result of the client following our advice, they lost about three hours working time but no data. The staff used this time to catch up with those “non-computer” jobs. We lost most of a nights sleep and the staff were able to just log in when they got to the office this morning.
As with all incidents like this we debrief after the event to see how we may change our proceedures in the future and consequently we have introduced a further check all Octagon engineers will carry out before rebooting servers.