Game

Improving Latency & Race Starts

If your league has been one of those affected by recurring stuttering races over the past few weeks, or worse, I want to start with a sincere apology. I’ve seen the same league names and race times coming up repeatedly in reports, and I’m very grateful to everyone affected for their patience.

I would not normally go into this much technical detail in a public announcement, but at this point I think you deserve a proper explanation. I also hope it offers some reassurance, because when this doesn't occur, we have already seen the platform run with complete stability, including a recent day with zero server issues in over 24h.

Technical cause identified

In technical terms, the problem over the last 2-3 weeks has usually come down to a single race hitting a rare simulator bug. When that happens, the simulator process crashes, consumes server memory, and can drag other services down with it. That makes it look like a wider server problem, when in reality it is more like one isolated race out of thousands causing a chain reaction.

This took a long time to trace because it leaves behind almost no useful evidence, no clear logs, no obvious error output, and very few clues beyond the fact that things break. I’m now very close to the root cause. It appears to be related specifically to races where refuelling rules are changed shortly before the start, leaving some managers preparing strategies with refuelling enabled and others with no-refuelling rules. That mismatch appears to be the source of the corruption. There are a few ways to solve this, and I’m working on those fixes now.

Alongside that work I'll also be moving into other simulator improvements like improving race starts, because those have not been smooth in recent weeks too. Thank you again for your patience while this is resolved.

And the race posts results after? Or it's actually not running when this happens?


This sounds very much like what my changes are intended to fix. Sorry to hear you have had this experience, that is obviously unacceptable.

This is my sole focus now. I am going to spend the entire day on this, because we are seeing the same issue recurring. It happened to one race last night which got stuck in that loop  that ended up making the service laggy for everyone.

I've just deployed an attempted fix to all live simulators, which is an educated guess at why simulators were crashing when lots of cars retired. It looks like retirements were the issue, not so much fuel.

A fix for fuel-related glitches was applied yesterday also, and it improved the situation but didn't resolve this fundamental sim problem.

Redundancy is being deployed at every level: all services that serve the sims, the servers themselves and the simulator core code.

I've also just deployed an efficiency improvement to web systems which should optimise the performance of the website. If you used to see it getting stuck loading sometimes this should be improved.

UPDATE: We just had our first sim crash under the new system. Hopefully, nobody noticed, except that one race. It looks like the system absorbed it as it should.

Now I need to find out why this race crashed at all.

Service has been very stable today, so the fixes applied yesterday seem to be working. I'm moving my focus to improving race starts and overall car movements.