Game

Unscheduled Maintenance (2017-07-26)

UPDATED 14:21 GMT.

Our sincere apologies for this unplanned interruption and maintenance. The game had been becoming progressively slower each night due to rapidly increasing surges in players and we have been logging and investigating the situation for a few days. Last night, during peak hours, we identified a bottleneck and decided to act immediately to improve the situation, as the game had become unstable to play. It caused a lot of disruptions to races, but this was better than letting it to go any further. Working overnight and into this morning, we have made substantial improvements to the service. We have scheduled further maintenance on the weekend (Saturday at 06:00 - 10:00 GMT) to do one final upgrade and we will be working throughout the weekend to make efficiency gains in the areas our research has identified this week.

Thank you to everyone for being so understanding and patient. We really have the best community, and it's growing unbelievably quickly. If you have had any issues that need admin assistance, please post here and we will be happy to help.

UPDATE 6 (13:00 gmt): Everything should be back to normal now. I will be dealing with individual cases throughout this afternoon.

UPDATE 5 (11:00 GMT): All leagues should now be postponed. Now proceeding to reset practice/qualifying for the races that have been postponed, then all should be back to normal. This may take a further couple of hours. Of course, I expect a lot of odd cases as well where people might have issues unique to them, and I'll move on to those after this.

UPDATE 4: Races are being postponed, but they are mainly going to the 28th rather than the 27th. This is because our postpone script is searching for everything in the period that may have been affected, and moving it to the first race slot at least 24 hours after that. So, it's looking for something at least 24hrs after the 27th. Hopefully that clears up any confusion about the new next race dates that are appearing.

UPDATE 3 (08:20 GMT): We are now in the process of postponing all affected races (this will fix the "Go to race" issue). Your league should return to normal in the next couple of hours. If you have had any other urgent issues, please post them here and we will investigate each case and try to help.


UPDATE 2: During the second maintenance window we were able to perform all the upgrades we wanted. However, taking everything down in the middle of running will have caused more problems as well, so we expect a long night and morning tomorrow resolving your issues. We will provide support and be happy to postpone any affected leagues.

UPDATE: In attempting to upgrade relevant services we discovered a previously unknown "cap" our server provider enforces that needs to be lifted before we can perform all of the upgrades we believe are necessary to address the situation. We will need a second maintenance window for that. Basically, they've tied our hands and only allowed us to half-fix it in this first window. We're on to them immediately with critical support tickets to raise the cap.

Any races that didn't run we will be happy to postpone. Over the next few hours we will have a lot of data arriving about which leagues didn't run etc. using our own internal tools that will make it easier to manage.

Again, apologies for the experience tonight, and all week really, we knew the situation was bad all week and were already monitoring it very closely. Tonight we caught several layers of what was causing the slow-down in the act and have immediately begun work on improving it.