Does GameLift Really Run Itself? (4) — Splitting the Backend and Updating a Live Service

03 Nov, 2026

In the last post, I described how we placed match servers in many locations with one fleet. Matches ran locally, and management ran at the center. Our game server also changed a lot while we built this structure. This post is about that change.

From one server to servers by function

The starting point is what I described in Part 1. The game server was one binary that ran on Windows with IOCP. An Agent that I wrote started and stopped the dedicated servers.

The single binary did not fit the structure in Part 2. The entry points had to be in many regions. The core logic had to be in Virginia. So we split the server.

First, we ported the Windows source code to Linux. For the network layer, we used io_uring instead of IOCP. io_uring is the Linux interface for asynchronous I/O. Then we split the server by function.

• PlayerEdge: the entry point where clients connect.
• Room server: manages rooms.
• Squad server: manages squads.
• Bridge server: coordinates GameLift and FlexMatch. This is a new server.
• Chat server: handles chat.

We also moved the dedicated servers to Linux. We ran them on AWS Graviton instances.

Splitting servers makes management more complex. The servers also need a new way to talk to each other.

Why we took Redis out

At first, we used Redis for communication between servers. But it caused problems with the way we used it.

We made every server call SCAN and GET every 30 seconds. As the number of servers grew, these requests piled up on Redis. Redis runs commands on a single thread. So we expected the same bottleneck to come back as the server count grew, even after code fixes.

Pub/Sub also had a problem. Redis Pub/Sub does not keep a message when no subscriber is connected. Messages that arrive while a server restarts are lost.

This was less a problem with Redis itself. Our usage pattern and the nature of Pub/Sub did not fit together. We took Redis out of the message path between servers and out of server discovery.

Kinesis Data Streams and DynamoDB

We filled the gap with Amazon Kinesis Data Streams and Amazon DynamoDB.

Kinesis Data Streams keeps messages for the retention period. So you can read messages again within that period. We save how far each reader got as a checkpoint in DynamoDB. Also, servers do not connect to each other directly. Every server connects only to Kinesis Data Streams. The N×N TCP connections between servers went away. To add a new server, you only connect it to the stream.

But Kinesis Data Streams does not deliver each message exactly once. The official documentation says producer retries and consumer restarts can deliver the same record more than once. The application must handle duplicates. (Handling duplicate records)

We put Kinesis Data Streams in Virginia. So lobby and management messages have some delay. We built the managers for these messages to tolerate delays of hundreds of milliseconds. The client and the dedicated server exchange real-time play data directly. So this delay does not affect real-time play.

We stored state in DynamoDB:

• Server discovery and heartbeats
• Player state
• Kinesis Data Streams checkpoints
• Records of squads and matchmaking tickets

We set up DynamoDB as global tables, with replicas in several Regions.

We built our own C++ message bus

This part needed our own work.

AWS provides the Kinesis Client Library (KCL). KCL handles load balancing across workers, failure handling, checkpoints, and changes in shard count. KCL itself is a Java library. It supports other languages through a Java-based daemon called MultiLangDaemon. (Kinesis Client Library)

Our servers use only C++. We decided to handle the stream directly inside the server process. So we built our own MessageBus. Here is what MessageBus does:

• It finds shards. It opens split shards in order.
• It saves checkpoints. After a restart, it reads again from that position.
• It filters out duplicates and puts messages in order.
• It monitors processing delay. When the queue is full, it waits.
• It shuts down safely.
• It can also connect to streams in other Regions.

Kinesis Data Streams did not do duplicate removal and checkpoints for us. Our code did. Building it ourselves also led to incidents. I will cover them in the next post.

What we change during live service

After we split the servers and connected them through AWS services, we could do more. We could change three things without stopping the service.

The first is the dedicated server version. Say matches now run on version 100. When version 101 is ready, we create a new fleet. Matches in progress finish on the old fleet. From the next match, we matchmake into the new fleet. We clean up the old fleet after that.

The second is matchmaking rules. When many players are online, we group players with similar skill. When few players are online, we allow a larger skill gap so matches start quickly. We changed FlexMatch rules based on player count.

The third is how we place match servers. We changed the latency policy of the game session queue. Take players in South America as an example. When there are few players in South America, they play in Dallas. When there are more players, they play in São Paulo.

GameLift did not make these changes on its own. We decided when to change what. We also designed the order of steps for a version swap. We put this process into our GM tool. We do not need to open the AWS console. GameLift and FlexMatch accept changes through their APIs and apply them. We built our tools and switch-over flow on top of those APIs.

Next post

Up to this point, things went well. But we went through a lot to get here. The next post covers six incidents from production.