Activenet distributed game server architecture

Last updated: 10 Aug 1998

Goals

We will deploy game server code to about ten Linux machines running all over the world. The end user will be able to fire up a game client and see all sessions available all over the world without caring too much about which server they connected to.

The implementation should be

Definitions

Strategy

The basic networking code used to propagate game information, called dptab, provides a way to publish tables of data. Each table has a name. Each data element inside each table also has a name. The names can be arbitrarily long sequences of arbitrary bytes. There are several predefined identifiers for use in table names, e.g. In addition, each game supported by Activenet has a 16 bit sessionType identifier. A sessionType is associated with a game name by adding a directory in the source tree under src/netshell/data with the sessionType in decimal containing a text file named name.txt, e.g. for Dark Reign, the filename is 666/name.txt. (To associate a localized session name, also create a sessionType.locale directory. e.g. for Dark Reign japanese, the filename is 666.jp/name.txt.)

Session identifiers (the same identifier is used within the SESSIONS and MYSESSIONS tables) are designed so that no central registry is required to generate them; a machine that wishes to create a session just does so. The identifier is the session's karma (a karma is a 16 bit number chosen at random) preceded by the session master's address. For instance, let's say the session host chose karma abcd. If using IPX, addresses are ten bytes long, so the session identifier could be 0.0.10.0,ff.33.ee.dd.ab.ce:ab.cd. If using IP, addresses are four bytes long, so the session identifier could be ee.dd.ab.ce:ab.cd.

Authentication

There is a single username/password database for all servers. Before connecting to a game server, call dpSetUsername(..., username, password). Password is currently ignored, but will eventually be used to answer challenges from the game server.

By default, the library acts as if dpSetUsername(,,,, "guest", "") had been called. This creates the anonymous (guest) user.

dpSetUsername remembers the username and password. Later, if a host or server wants to know who we are, they send us a challenge consisting of twenty questions about our password, and we respond with the username and the answers to those twenty questions.

Logging in to Game Servers

When dpSetGameServer() is called, it sends a login request to the gameserver. When the gameserver receives the login request, it may authenticate the user as above, or (for guest users) just accept the user. It indicates that the user has been accepted by creating a record for the user in the HOSTS table, and publishing that table to the client.

Session data propagation

Each game server publishes a SESSIONS table which is read by clients that connect to that server, and lists all sessions available worldwide.

Each game server also publishes a MYSESSIONS table which is read by other servers, and lists all sessions started by the game server itself or by clients attached to it.

Each game server mounts the MYSESSIONS tables from all the other game servers onto its own SESSIONS table. The data from the individual MYSESSIONS tables combines to form one giant table.

Each game client publishes a MYSESSIONS table which is read only by the game server it attaches to. A game client that creates a session is called a game host, and is the initial master of that session. To create a session, a machine simply puts a new record in MYSESSIONS. The new session should show up in the SESSIONS table received from the game server within five seconds.

Each element of the SESSIONS and MYSESSIONS tables is a dp_session_t.

Creating and Joining Sessions

When user code calls dpOpen to join or create a session, it creates the following tables: Then, if joining an existing session, dpOpen adds the session master as a publisher of the HOSTS.SESSIONID and PLAYERS.SESSIONID tables. It then sends the session host a join request containing the id of the desired session. dpReceive() on the host then creates a record in his HOSTS.sessionid table for the client, and, if the client is not itself, it publishes the HOSTS.SESSIONID and PLAYERS.SESSIONID to the new client, and subscribes to the new client's MYPLAYERS table. Finally, dpReceive() on the client publishes the client's MYPLAYERS table to the master, and signals the user code via a callback that the session join was successful.

Creating Players

To create a new player, a client picks a two-byte player id from the range given it by the master in its HOST record, then creates a player record in its MYPLAYERS table. When the master notices the new player appear, it may verify the player id, and it adds the new player to the PLAYERS.SESSIONID table.

User data

The profile for a particular user is stored on the server in table PLAYERINFO by userid. Userid's are strings assigned by the user but checked at the central database for uniqueness during the account generation process. A filter can be used to subscribe to PLAYERINFO and retrieve just a particular user profile.

All game servers subscribe to the master server's PLAYERINFO table.

Reducing amount of data sent

The servers will maintain a table called SESSIONINFO which lists summary information for each session type. This relieves the client of the need to enumerate all sessions when all it wants is to give the user some idea of where the action is.

Table subscriptions can be filtered. The publisher only sends changes in records that match the filter. Initially, only two filters will be supported: MEMCMP(offset, len, value) and !MEMCMP(offset, len, value). That allows clients to subscribe to SESSIONS with a filter for, say, I'76 sessions; they might mount that subscription at SESSIONS.I76 (where I76 is the sessionType value for I'76). It also allows servers to subscribe to another server's SESSIONS table, and specify "don't give me sessions that have my IP address in the game server field".

When the session master goes offline

When any participant in the session loses contact with the initial session master, it switches its subscription for that table to the machine of the player with the next player id. If a machine discovers that it is now the master, it places a record in MYSESSIONS, which automatically informs the game server of the change. If the original machine comes back online, it should notice that the game server lists a new host for the session, and reenter the session as a client rather than a host.

Note that the initial session master need not create any players in the session. This could happen in the case of a game server. We might want to set a bit in the session record for these sessions saying "session cannot migrate". In that case, if the server went offline, the players could continue playing, but nobody could join (or leave!) the session.

Firewall Strategy

To accomodate port mapping firewalls, the address used on IP has been extended to six bytes to include the port number; clients no longer always open a particular port.

To allow clients to be behind SOCKS firewalls, the guidelines for SOCKS clients will be followed on clients (but not on the server).

To allow the use of IP addresses in universal client identifiers, clients which are behind a firewall will ask the game server what IP address and port number they appear to be using, and will use that when constructing ID's. Client machine records will contain both the external IP adr/port number and the internal IP adr/port number. When a client opens a connection to another client, it checks to see if it might be behind the same firewall; if so, it pings the other client at both addresses, and uses the address/port number of the first reply to open the connection.

Server Login, Buddy Lists, Instant Messages, and Score Retrieval

The Login Process

See tserv.h.
  1. The client opens a connection to the server by calling dp_setGameServerEx().
  2. The next step depends on whether the user has an account already: When dpio_get() returns a packet that starts with dp_TSERV_PACKET_ID, dpReceive() passes it to tserv_handle_packet(..., &event).

    When something interesting happens, tserv_handle_packet will return dp_RES_OK, and event will contain the result to the pending operation (login, create, or activate). dpReceive will convert this to a packet of type dp_ACCOUNT_PACKET_ID.

  3. If the account exists but has not yet been activated, the server will reply with an event telling the client to prompt for the secret activation code; the client prompts the user for the activation code, then calls dpAccountActivate(dp, secret_code) aka tserv_account_activate(tserv, secret_code)

How the UID's of all players in a session become known and trusted

What the client can find out

The following info is obtained by subscribing to particular tables on the game server: The user code accesses these by requesting object deltas for those tables.

Scores

Score data is handled by the modules in src/score, and also by server/servscor.c (for the server-only code) and dp/dpscore.c (for the client-only code). Clients send score reports up to the server whenever a player leaves. The server cross-checks the reports, and uses them to update a score database, kept in a standard database system (at the moment, gdbm) for easy querying by other software.

The server maintains one table, dp_KEY_MYSCORES, which all the clients publish score reports into. It also maintains one score table per session type, [dp_KEY_SCORES.sessionType], containing the cumulative scores for each user of that session type. When it receives a record in the dp_KEY_MYSCORES table, it cross-checks it, and eventually updates the scores database and the [dp_KEY_SCORES.sessionType] table.

Likewise, each client maintains two tables, dp_KEY_MYSCORES (published to the server's dp_KEY_MYSCORES) and [dp_KEY_SCORES.sessionType] (subscribed from the server's [dp_KEY_SCORES.sessionType]).

User code stores scores by calling dpReportScore2() whenever a score changes; the current scores when a player leaves the game are automatically placed in the client's dp_KEY_MYSCORES table and uploaded to the server.

User code retrieves cumulative scores by calling dpRequestObjectDeltas([dp_KEY_SCORES.sessionType]) once; this triggers a message any time a new score is available. Later, there will be a way to specify which range of users you are interested in scores for.

Behind the scenes, a central score server gathers score data from all servers continuously via http, sends the score data through a game-specific function, and makes the processed score data available again via http. The game-specific function (and the whole master score server) is written in Java. Two of these will be run, and will produce identical results, so game servers can connect to either one if there are backbone problems. In fact, game developers can run their own little score server while developing the game-specific server-side score processing function; when the function is done, it is uploaded to the production score servers, which look in a file scoreNNNN.jar for the score-processing functions for game type NNNN.

Data Sharing between Servers

Originally, the dptab subscription mechanism was designed to allow servers' session tables to be published to one or more master servers; the game servers would then subscribe from the big session tables at the master servers. If a connection goes down and is reestablished, currently, the entire table is retransmitted.

For large nonvolatile tables, we probably want to avoid that table retransmission. A new method, based on http, is used to share crash data between the game servers and the master crash server. A similar method will be used for score data and possibly password data.

This method will probably be built into dptab, along with a 2nd persistance mechanism which backs tables with a .gdb database file on the fly rather than at dptab_freeze time.


Dan Kegel