Datacentre outage

UPDATE (18:53 GMT).

We believe the issue is now resolved and no longer see any traces of it throughout the service.

ORIGINAL MESSAGE:

There is currently a service outage at the datacentre hosting race.igpmanager.com. We've been told (by the datacentre) that this may take several hours to address as the issue is a multiple harddisk failure on their end, with all requiring replacement.

We're very sorry about this, as it is causing races not to run, although igpmanager.com is still available and unaffected.

We are currently working on a temporary solution to get races running again tonight without the servers we normally have at our disposal.

The datacentre have confirmed to us that the issue is a hardware failure on their end. They will need to replace the hardware and we will need to restore a backup to the server once they are done.

We appreciate your patience at this time.

The datacentre has updated their estimate to "several hours" of downtime, which is unacceptable.

I am looking in to what possibilities exist to create a temporary setup to run the race service from igpmanager.com. This may take a while, but should provide some relief and a temporary alternative until they fix their servers.

We are making good progress on a temporary solution (temporary service) to replace the offline race.igpmanager.com server. Our target is that races this evening and tonight will run normally, even while the normal service is offline.

No, sorry. As mentioned in the OP, if you're reading this then it's already too late!

Our temporary solution may be in place for the 7pm (British Summer Time) races, and we doing all we can to achieve that, but in many areas we are simply being held back by how long it takes to copy files etc. not how quickly we are working. When working on this scale you can spend an age waiting for files to move from one server to another.

We have our new solution in place, and races are beginning to run.

If your race was supposed to start 3 minutes ago (on the hour) hang around. We'll get it going shortly. :)

Unfortunately it looks like it's going to be tough to get the 6pm races running, my original forecast of 7pm (BST) is more accurate. We thought we had everything ready to go now, but it's not that simple.

We are still working to get the 6pm races going, but if they have not started by 30 minutes past the hour we will postpone them and refocus our energies on getting everything online for 7pm.