Does GameLift Really Run Itself? (5) — What Went Wrong in Production
In the last post, I explained how we split the servers and changed versions and rules during live service. But it did not go well from the start. This post covers six incidents from production. For each one, I describe the symptom, the cause, and the fix. Most were not problems in GameLift or AWS. They were places where I misunderstood how something worked.
1. A new matchmaking rule did not take effect
Symptom. We wrote a new rule. If not enough people joined, the rule waited five minutes and then filled empty slots with AI players. But for a while, players got matched with AI in less than 30 seconds. We had uploaded the new rule, but real matches still ran on the old rule.
Cause. Our requests to replace the rule set were being rejected. A FlexMatch rule set can define team names. I gave each rule set different team names. The official documentation explains what to watch for when you change the rule set of a matchmaking configuration. If matchmaking tickets are in progress, a new rule set with different team names or a different team count may not be compatible. For this kind of change, the documentation says to create a new matchmaking configuration. (Edit a matchmaking configuration)
Fix. After that, we used the same team names, blue and red, in every rule set. From then on, rule changes during live service took effect without problems.
2. A match with mostly European players ran in Seoul
Symptom. Several European players and one Asian player ended up in one match. This kind of group happens naturally. But the match ran in Seoul, not in Paris or Frankfurt. For the European players, Seoul was the farthest option. The players measured ping correctly. We collected a lot of logs and spent a long time on analysis to find the cause.
Cause. It was my mistake. The client measures ping to more than 20 locations. I thought that was too many values. So I made the client report only the 10 fastest.
The European players' lists did not include Seoul. Seoul is not in the 10 fastest locations for players in Europe. The Asian player's list had Seoul as the fastest location. The result looked as if only the value from the player who reported Seoul counted. That value was the lowest of all reported values. So Seoul looked like the best candidate.
I could not confirm exactly how the service handles missing values. One thing is certain. If a sample leaves out the high pings, you cannot compare candidates fairly.
Fix. Instead of the 10 fastest, the client now sends all measured pings for the open locations. After that, matches were placed in the location that was best for most of the players. Note that the UDP ping beacon documentation from Part 3 says not to include latency for locations without a fleet in matchmaking requests. That is why we limited the scope to "open locations."
3. Players got a match but could not join it
Symptom. This happened during Steam Next Fest. Matchmaking worked, and a match server was placed. But some players could not get into the dedicated server.
Cause. When a match is made, GameLift reserves a player session for each player who will join. A reserved player session is in the RESERVED state. The player must connect to the match server within 60 seconds. The match server must also validate that player's entry (AcceptPlayerSession). A connection alone is not enough. If this validation does not finish within 60 seconds, the session goes to the TIMEDOUT state and the reservation is released. (describe-player-sessions)
We added a short presentation after matchmaking. We also gave players plenty of time to pick characters. So when a player took a long time to pick a character, the 60 seconds could run out before the match server validated the entry.
Fix. We could not force players to pick faster. So we changed the flow to get a new entry reservation just before the match starts. We did not extend the 60 seconds of the existing reservation. The 60-second limit stays the same. We got a new reservation just before the match starts, so players have enough time to join. After that, this problem did not come back.
4. I treated a cancel request as a completed cancel
Symptom. Players cancel matchmaking more often than you might expect. But sometimes a match went ahead for a player whom our server had already cleaned up.
Cause. I sent the cancel request and then cleaned up on the server side right away. I assumed the cancel would succeed. But sometimes FlexMatch had already made the match. In that case, the cancel is not accepted, and the match goes ahead. I had already cleaned up the player. I had not considered this case at all.
Fix. We separated the cancel request from the cancel confirmation. We end the matchmaking search only when the cancel is confirmed. If a match was already made, we send the player to that match.
5. Messages backed up even with more shards
Symptom. From here on, the incidents are about Kinesis Data Streams. We use Kinesis Data Streams as the communication channel between servers. If it has a problem, the whole service shakes. We created many shards. We also created many threads to send and receive. But messages still kept backing up.
Cause. It was the HTTP client configuration in the AWS SDK for C++. The default value of maxConnections, the maximum number of connections one client opens, is 25. (AWS SDK for C++ client configuration) I did not know that I had to change this value. No matter how many shards and threads we added, those 25 connections were the bottleneck.
Fix. We raised the maximum number of connections to 150. We also set the thread pool to 150 threads to match. After that, this backup did not happen again. I learned something from this incident. When a part we built ourselves has a problem, I should suspect my own configuration before the infrastructure.
6. Many servers shared reads on one shard
Symptom. When we added servers, requests to fetch messages started to fail.
Cause. There are two ways to read data from Kinesis Data Streams. One is to poll with GetRecords. The other is Enhanced fan-out. At first, I used only GetRecords.
GetRecords allows up to five calls per second per shard. (GetRecords) All servers that read that shard share this limit. We attached several servers to one shard. Each server read once per second. So whenever more than five servers read one shard, reads failed.
Fix. We split streams by message type. We applied Enhanced fan-out to the paths that need fast responses. Enhanced fan-out is not a special kind of shard. It is a way to consume data. Each consumer gets its own throughput per shard, and the service pushes records to the consumer. (Enhanced fan-out)
Looking back
Let me go back to the question in the series title. Does GameLift really run itself?
In many ways, the answer is yes. GameLift ran our dedicated servers and restarted processes that had ended. It scaled capacity by the rules we set. FlexMatch and the game session queue grouped players and chose where matches ran. Because of this, a small team of server programmers could run a global service.
But all six incidents above came from our side: rule set team names, the scope of the ping list, join time, cancel handling, SDK configuration, and the read method. The services worked by their defined rules. Understanding those rules correctly and filling them in for our service was our job.
So here is my answer. GameLift did the work we gave it. We decided what to give it, and we checked that the results matched our intent. We still verify those results ourselves and make them more efficient today.
Thank you for reading this series.