Backup process complete

Okay. After the wide range of issues we had yesterday with the server upgrade we had no alternative but to roll back 24 hours to a backup from the point at which the server was put live. So we have effectively gone back to data from November 12 at around 05:00 GMT.

All races that ran yesterday (or this morning) have been postponed and reset to the next available slot. Any races starting later today will run normally. Check your league schedule for the next start time for your league.

We have addressed the issue with the server overnight and everything will run as normal from this point onwards. The server migration we carried out yesterday / today was a huge operation that moved iGP to a totally new datacenter and as you can imagine there were many technical challenges involved.

The server migration was carried out by a different team of people to the team that handle the iGP websites, and there was some disconnect between the two. The people handling the migration disappeared after the operation was complete and we were left with a malfunctioning service. We then had to wait for them to return to rectify the situation which is why the issue was prolonged and affected so much data. We concluded that the most sensible option was to revert to a backup to remove the corrupt data, and take the site offline until they returned and rectified the issue, so that nobody wasted any more time entering data that would be wiped away. Our apologies for any inconvenience this may have caused, and we appreciate your understanding and patience on the matter.

For anyone interested in what the technical problem actually was yesterday: Database operations that wrote to disk were performing very poorly. This is why operations that read from the database (browsing the site) performed well, but processes that wrote to the database (inserting results, updating results etc.) were very slow and/or contained errors. But now we can continue from a clean slate, the bad data is gone and the underlying issue has been fixed.

Just to reiterate this is the last such planned upgrade, so you don't have to expect any more drama like this. We should have enough hardware now that further upgrades won't be necessary for a long time. The reason for the recent string of upgrades (this is the third one) is to prevent "Connection problem" errors that happen a couple of times a week at peak hours. The aim is to have eliminated them almost entirely now.

Soon we will begin work on addressing lag and "warping" in the viewer, which also only surfaces at peak hours typically. We have a planned solution for this but it will take some time to develop.

Everything was just one thing - the issue with file write speeds. Let me address this example (reputation) as a demonstration:

Because it took so long for the results to be written to disk, the file would cease execution mid-way, then run again, and the cycle would repeat 2-5 times until the system was satisfied that it had completed executing. However, as it parses the data the file would also inject reputation and other rewards alongside the data, 2-5 times over, resulting in huge rep gains, and other issues such as multiple race contract expirations etc. Everyone got a positive gain because when you add negatives together it creates a positive effect.

This is really the breakthrough we've had and the reason we're upgrading the server. Issues along these lines have been occurring for some time, and creating lots of headaches for everyone. Initially we went chasing after each individual issue assuming there was something wrong with contracts, something wrong with the reputation calculations, something wrong with the simulator (when it throws a connection problem) etc. It took time to discover the root, because the issues seemed random and unconnected at first. Every specific example of something going haywire yesterday can be traced back to this single root cause. And if you go back further to when these things have occurred in the past they have similar causes that were a result of the server not being powerful enough (which we've addressed in this upgrade). So, we're hoping that as of today we can put all this behind us and have a much more reliable and stable system in general.

We did, through Facebook, Twitter and a news article on igpmanager.com (also visible on the in-game home page).

If you mean we should have advised that we'd have to roll back 24 hours, I'm afraid we didn't get much advance notice on that one either. It was obviously not part of the plan. I hope this thread has sufficiently explained that and why it was necessary.

Out of interest though, is there any additional way you would like it to be communicated? Or was this just a misunderstanding that the rollback was not a planned part of the work.

The data is in on several races that have run now, and has put a wide grin on my face. We've gained a tenfold increase in performance from the new server over the old one, comparing execution time of race results. This is what we were looking for, minus yesterday's issues.

This final (and much more complex upgrade) has yielded a performance gain a magnitude greater than the last two combined. I know it's almost meaningless on the front-end, things will still essentially run the same, but from a technical standpoint this is significant. This should make a noticeable difference to the stability of things.