Solved

Race problems 29 Dec 20:00 GMT

After reviewing what happened, we worked out a way to prevent it and implemented it immediately.

Not to downplay anything (we will always work to improve everything we do), but there is an inevitable exaggeration of even the smallest fault that comes hand-in-hand with scheduled league races. By that I mean: we can provide a flawless service for a month, then have 2 nights in a row where there is a brief 20 minute disturbance, let's say occurring at 8pm exactly. Each night it's a totally different source. Unlike any other game, anyone who races at 8pm will now believe there is a "constant" problem happening "every day" or "all the time". Yet on paper, that equates to 98.7% flawless service in a 24 hour period, and 99.91% in the period of a month. For any other service, those would be very respectable numbers. For a service where people need to operate on a schedule, it can be extremely disruptive.

I know none of that matters to anyone except us, but thinking aloud it's an interesting challenge to deal with as a developer, one which I don't think many games have. When we have a problem at the same time for a couple of days in a week, we know how that's going to look to those people, and it just amplifies the whole situation. I offer this up merely to give some perspective, and show that on paper we are competent (for those who clearly question that and suggest how we do our jobs). We also understand that we can't have issues like this occurring, period. We're always working towards 100% up-time and all of the money from the game gets re-invested into development of features and systems, much of it to ensure that.

It has been fixed each time. Bear in mind, you can only see the end result and not the cause (databases, file systems, networks, servers, code etc.). It might be helpful to explain by analogy...

Let's say your car had problems with the indicator not blinking. So you go to the garage and they replace a faulty bulb, and it's fixed. A few days later it happens again, because of the wires that lead to the indicator, not the bulb. They fix the wires. A couple of days later it goes for a third time. It turns out the computer in the car that manages the electronics has a fault, and it's nothing to do with the light bulb or the wiring, which are both working from the previous visits. Each time the indicator wouldn't blink and was also genuinely fixed.

That has parallels with this week. MMO server systems at scale are made up of interdependent components, where what might manifest in the same outcome multiple times has a different cause. In each case it has been resolved.

Also, a quick reminder of what I wrote above for perspective: 99.9% up-time over the course of a month is respectable for any company in the technology industry. I also completely understand if the 0.1% of faults lands on your race several times in the space of a week, the experience is the inverse of that. That's the unique challenge we face. We can be delivering overall performance up there with the best in the industry, but due to scheduled gameplay, some people occasionally experience the inverse of that.

It's a tough challenge, but that's what we've signed up for, and we're doing our best to improve what we can. I appreciate no more explanations are really warranted or needed at this point - you just want results. Me too. :)