Official

IMPORTANT: Memory leak [DEV]

To my knowledge, this is the last serious stability issue to resolve with the update, so once this is resolved, we're back on track.

We have had an intermittent server-side memory leak appearing for ~48hrs, which has only been resolved by me manually clearing out the races that cause it. The effects when it happens are exactly what has been reported: laggy viewers for those connected, some races not starting on time etc. When it happens, it impacts all races on the same server, and only that server, not the whole service. During peak hours every server gets a lot of races, so it can become exaggerated.

All the reports keep recurring on the same tracks (Abu Dhabi, Netherlands, sometimes USA seems problematic too). So, I have re-exported these tracks clean with some hopeful fixes and they have gone straight to the live service as an urgent fix attempt. I will keep monitoring the situation.

An insight I want to give everyone as well is what it's like on my end in these scenarios, so you can understand the efforts being put into resolving them. If I'm so much as 10 minutes late to my desk to stop a memory leak, that's as good as not showing up at all for most of you, because your race either starts late or not at all, or becomes unplayable for 10 minutes if you can get in. I'm also a dedicated parent and have to look after my children. Combined, this can create a perfect storm at times, where not having 24/7 availability or a 10 minute delay here or there is enough to set an impression with you that nothing has changed or nobody cares. I want to assure you I care very deeply about resolving these issues and am devoting a lot of time to this.

My availability is much higher from the time of this post, for the next week, I'm fully back on iGP with my full focus. I'm going to be working long days and patching everything up to get this year off to a strong start.

STATUS UPDATE
20:00 UTC: All the 8pm races have started with the newly exported tracks and I am not seeing any signs of memory leaks. This is a promising start. I will continue monitoring in real-time. An hour ago, there would absolutely have been systems flashing alerts at me with this many races starting.

20:00 UTC is the absolute peak hour, so it's a relief that I'm here for this one. Apologies to anyone at 19:00 UTC who experienced issues, I was 10 minutes late for your slot and the newly exported tracks were not live yet. Some of you did experience quite bad lag or in rare cases races not starting.

STATUS UPDATE 2
20:40 UTC: Half of the 8pm races are now complete, and there has not been one issue flagged with the new tracks, no memory leaks. It still looks solid, so I'm going to roll out the rest of the tracks in the same format. They should resolve some remaining visual issues with cars stopping too early in the pit lane as well.

I understand your frustration, and you’re absolutely entitled to feel the way you do. When races aren’t running properly, it’s not enjoyable, and that’s a fair reaction.

I do need to be clear on one point though: rolling back isn’t an option. The update involved a fundamental change to the underlying data structures, and reverting would introduce significantly more problems than it would solve. In situations like this, the fastest and safest route is stabilisation through targeted fixes, not reversal.

That’s the approach we’re taking. The issues seen over the past two weeks have largely come from edge cases exposed by a large migration, rather than from the core direction of the update itself. Those are exactly the kinds of problems that get resolved through focused patching, and that’s what’s been happening. We’re already seeing stability improve as those fixes land, even if that’s not immediately visible to everyone yet.

Looking ahead, the update lays the groundwork for some meaningful improvements to iGP this year. Rather than dwelling on the disruption of the past couple of weeks, my focus is on demonstrating that progress through concrete changes and continued fixes. I’ll be publishing a roadmap shortly so it’s clear what’s coming next and where effort is being directed.

Yes... we had a new memory leak (first one in ~6 hours). I rolled out an automated service designed to catch the issue and stop it in its tracks, and regrettably it restarted the services / simulators, which is why your race restarted.

This was not intentional, so I've taken the automated system out again and will review what happened. Apologies, as you can see I'm still working towards a permanent solution. I'll step away from it now and resume tomorrow. Essentially, I'm trying to set up a system which can automate my role, so the chain during a fault won't depend on human input, as it will recover itself. Once such a system is online and stable, the improvement in the overall quality of the service will be significant. There will be 0 response time for memory/cpu related faults once it's working and it will isolate the impacted races, not allowing them to impact a wider server in future.

Thanks for reporting it. If you'd like to re-run the race simply put in a request and we'll take care of it.