Sourcewalker — English

Does GameLift Really Run Itself? (3) — Placing Game Servers Across Multiple Locations

In the last post, I described how we brought the backend together in Virginia. But we should not run the matches in Virginia as well. Real matches must run close to the players. This post is about how we placed those match servers.

We did not create a fleet per region

In Amazon GameLift Servers, a fleet is a set of resources that runs game servers. A fleet has one home Region. A multi-location fleet can add other Regions or Local Zones as remote locations. You can also set capacity separately for each location. (Amazon GameLift Servers service locations)

Let me define two terms here. A Region is an AWS geographic unit. A location is a place where a fleet actually runs servers. A location can be a Region or a Local Zone.

We did not create a fleet for each region. The home Region of our fleet is Virginia. When we build a new dedicated server, we upload the build to GameLift in Virginia. We create a fleet from that build. Then we add several locations to the fleet.

GameLift then launches instances in each location. It installs and runs the dedicated server on those instances. We do not have to repeat the same work in each region.

"One fleet" comes with a condition. It does not mean the whole service always has exactly one fleet. When we replace a version, the old-version fleet and the new-version fleet run together. It means we did not create separate fleets for each region.

How the match location is chosen

Here is the flow from a matchmaking request to joining a match.

  1. When a player connects, our server sends the list of locations that are currently open.
  2. The client sends UDP pings to each location. It measures the latency.
  3. When the player requests a match, this latency data is sent to FlexMatch with the request.
  4. FlexMatch builds a group of players that fits the matchmaking rules.
  5. The game session queue takes that group and places a match server.
  6. Our server receives the placement result through Amazon EventBridge.
  7. Our server sends the connection details to the client.
  8. The client connects directly to the dedicated server and plays the match.

Step 2 has one important point. We use UDP pings, not ICMP pings. GameLift runs a UDP ping beacon in each location. Most game servers communicate over UDP. So latency measured over UDP is closer to real game traffic. (UDP ping beacons)

Steps 4 and 5 make different decisions. FlexMatch decides who plays together. The game session queue decides where they play. This difference comes back in an incident in Part 5.

Placement follows the rules you set

When the game session queue has latency data, it looks for the location with the lowest average latency across all players. If you set a player latency policy, the queue does not place sessions where latency is above that maximum. You can also relax this limit in steps over time. (Player latency policies)

So placement does not always give the lowest ping. It works within the policies and priorities that we set. The location must also have real capacity left. There is also no promise that matches spread evenly across regions. The result depends on where players are and on the rules we set.

We adjusted locations based on usage. We removed locations with very little use. When we saw a region with many players but matches in a far location, we added a location. We made these decisions ourselves. Telling the client about the changed list is also our server's job.

Matches run locally, results go to the center

Matches run in many locations around the world. But we collect match results in one place.

When a match ends, the result goes to our web application server (WAS) in Tokyo. That server builds play statistics. It sends the information needed for the next match to our central servers in Virginia. In Virginia, we recalculate each player's MMR (matchmaking rating) and squad data. We sent this data through Amazon Kinesis Data Streams.

For this kind of result data, a round trip between Regions can take tens to hundreds of milliseconds without a problem. That is different from real-time play.

To sum up: matches ran in many places around the world. Match results and account management happened in one place. We managed one center ourselves. GameLift ran and managed the servers spread across regions.

Next post

After this change, it became hard for one server to handle everything. The next post is about splitting our single game server by function. I will also explain how we change server versions and matchmaking rules during live service.