Sourcewalker — English

Does GameLift Really Run Itself? (2) — From Regional Services to One Global Service

In the last post, I explained why we chose GameLift. This post is about our service structure. To explain match server placement, I first need to show how our backend changed.

Three regional services during CBT

During the closed beta test (CBT), we split the service into Asia, Europe, and the Americas. Each region had its own backend. Each region also had its own message stream. We also kept separate dedicated servers and Agents in each region. We were literally running three copies of the service.

After we analyzed the CBT data, we changed direction. This was before Steam Next Fest.

Continents are not network boundaries

Splitting by continent did not match reality. The Middle East was one example. By continent code, it is part of Asia. But we heard that its network path to Europe was better. So we had to write exception rules that sent some countries to the European servers.

Those exception rules were patches. A continent code does not tell you the real network distance or conditions in each area.

The operations load was also heavy. With a split by continent, deployment, configuration, and incident response all happen three times. Each region also has different peak hours. So someone had to watch one of the regions all day.

We asked a different question

I changed the question. Do we need a separate service for each continent? What if we keep the core logic in one place? What if only the entry point where players first connect is in each region?

With this idea, we redesigned the servers.

Aurora Global Database: write in one place, read in many

For the database, we used Amazon Aurora Global Database. Aurora Global Database takes writes in one primary Region. Secondary clusters in other Regions are read-only. (Amazon Aurora Global Database overview)

We put the writer in Virginia. Then we created read clusters in Seoul, Frankfurt, Oregon, and Singapore. Reads go to the nearest read cluster. All writes go to Virginia.

This structure has a clear cost. Writes take longer from regions far from Virginia. Also, a read cluster can only see data that has finished replicating.

Read at login, write asynchronously

So we also changed how the servers handle data.

When a player logs in, the server reads the data the player needs in one pass. Then it loads that data into the memory of the player's session. In effect, the server acts as a session cache for that player. Database writes go out asynchronously.

Think of a logistics center. A logistics center moves popular items to regional warehouses ahead of time. We do the same thing at login. We move the data a player needs into memory on the regional edge ahead of time. So we do not travel far every time we need to read a value.

Of course, this had problems too. At peak times, writes sometimes waited several seconds in the write queue. Aurora does not prevent slow writes for you. We handled this delay in a write queue manager that we built.

Real-time play stays on the dedicated server

At this point, you may worry. If a player in Seoul sends requests to Virginia, won't the game feel slow?

Real-time play data does not take this path. During a match, the client and the dedicated server exchange input and sync data directly. The messages that go to Virginia are lobby and management messages. These messages can take tens to hundreds of milliseconds without a problem.

But this structure only brings operations together into one unit. It does not remove the latency between Regions.

The question that remains

We brought the backend together in one place. So where do the actual matches run? Do we need a separate GameLift fleet for each region?

The next post is about how we placed match servers in many locations with one fleet.