State Synchronisation System for Networked Multiplayer Games

perchmysteriousΔιαχείριση Δεδομένων

30 Νοε 2012 (πριν από 4 χρόνια και 4 μήνες)

174 εμφανίσεις

Koku: State Synchronisation System for Networked
Multiplayer Games

Chris Dennett
, Prof. Madjid Merabti

Dr Abdennour El Rhalibi

School of Computing and Mathematical Sciences

Liverpool John Moores University,

Byrom Street, L3 3AF, UK


, A.Elrhalibi

Dr Marc Price

BBC Research and Development

BBC Research, Kingswood Warren,

Tadworth, Surrey, KT20 6NP, UK


In this paper, we talk about the conception and development of a
multiplayer state

system named Koku, a state
oriented framework developed for use with Java
based games that
is intended to pave an easy route to multiplay
er game creation.
The system has been inspired by the classical server
pool model
which is used in MMORPG games.

Categories and Subject Descriptors

H.4.3 [
]: Language Constructs and Features

data types, polymorphism, control structures.
This is just an
example, please use the correct category and subject descriptors
for your submission

The ACM Computing Classification

General Terms

Algorithms, Management, Measurement, Documentation,
Performance, Design, Economics, Reliability, Experimentation,
Security, Human Factors, Standardization, Languages, Theory,
Legal Aspects, Verification.


s are your own designated keywords.



state synchronisation system is intended to allow a
networked application to be created without requiring a deep
underlying knowledge of the network, or
requiring a complete
knowledge of how to dev
elop a large scale system that fully
exploits this network.

To utilise this system, it is necessary for the programmer to create
the game using purely state
based principles. A
lthough the
cation may use more resources through doing this, we

believe that computers are powerful enough these days to support
based management, although with larger applications the
complexity of this management may increase exponentially.

We wish to use Java for developing this system, simply because it
ws us to demonstrate our system concepts which can be ported
to other platforms in the future.

This is similar to the work being undertaken by other projects
such as Project Darkstar
, but we aim

to provide a different
implementation of the same principles that should scale more
efficiently by reducing inter
node bandwidth requirements
through state prediction mechanisms. For this, we need another
framework structure that handles our game state in

a different
way, but which may reuse some concepts from Project Darkstar
and other projects.

In principle, it is possible to use the hierarchical nature of the state
synchronisation network to filter certain types of data, but also to
reduce the accuracy
of certain types of state data as it travels up
the tree, cutting the data off a certain point. For instance, player
information contains both the ‘zone’ a player is in, as well as their
current position. As we go up the node tree, we find that the
n information becomes more and more redundant, so it
makes sense to restrict the transmission of this data and allow the
nodes to guess more, or prevent the storage of this data at higher
levels. For instance, the server does not need to know the position
of characters in ‘zones’, but may need to know what zone they are
in for administrative purposes. For instance, allowing the master
server to authorise user connections to zones.

In the general case, we assume that servers are separated and do
not run on t
he same machine for security, reliability, efficiency,
and natural zoning reasons. However, we may consider this in the
future as other state
synchronisation frameworks become more
core friendly



The main aim of the state synchronisation system is to share all
related data across the network, such as level layout. We
also need to share more dynamic elements of the state such as
information about the characters of a game, and prov
ide a model
of handling input and allowing the game state servers to modify
their state based on this input. This requires a comprehensive
method of state storage which is both efficient and can be easily
tracked and referenced, but also an event storage a
synchronisation mechanism. Emitted interaction event data is
useful to the server that processes the event to change the state of
the server, but also to the client which emits the data for
backtracking and working out where to backtrack the state data

Alongside this, we must feed information back to clients when
assumed that the variable value deviation in the client state
becomes too great to create a good future state based on their
version of the state without additional information. It is possib
to provide the prediction facility for certain types of quickly
replaced stream data like character movement or rotation, but it
may also be able to apply this concept to predicting when certain
complex game events that do not strongly affect game
nteractivity. It may be desirable to alter the event stream directly
with fake events to maintain interactivity.



Traditionally, games have provided their own synchronisation
logic in their various engines, usually with tight integration wi
the game code. Although this may creates a highly optimised
solution, the system is hard to initially develop, becomes difficult
to maintain, and is not directly portable to other games. However,
many interesting solutions have been developed in the pas
t to
counteract the problem of the effect of latency upon multiplayer
gaming and high network usage.

One of the first instances of state synchronisation in gaming can
be traced back to MiMaze, which uses distributed synchronisation
mechanisms to ensure tha
t each client has the same game state,
but with each having their own view upon this state
MiMaze is essentially an experiment to p
roduce a distributed
version of iMaze, a multiplayer game which has a centralised
architecture and works well in low
latency and low
environments, but has problems when used on the Internet

The server of iMaze is in charge of managing the sessions of the
clients, and sending the game maze configuration. Also, it
receives input from clients, alters the state based on this input, and
sends back the new game state through unicast
UDP. Each client
of iMaze is nothing more than a thin
client that communicates
player input with the server (input) and displays their view on the
global state (output). Also, they manipulate their connection to the
game session by joining and leaving the

MiMaze is distributed at both the computation level and
simulation level, and uses RTP (Real
time Transport Protocol) for
efficient communication. There is no ‘global state’
with MiMaze;
each client computes their own state from their own state and the
data received from other clients.

The state consistency maintenance method used in MiMaze is the
called ‘bucket synchronisation’ method which was designed
and developed in Mi
Maze itself
, and was based on a
simpler technique known as ‘lockstep’. It works by grouping
events into buckets aligned

on discrete time steps. Any events
received from other players are grouped into the bucket The
bucket synchronisation mechanism requires extremely fine
grained and accurate time synchronisation to function properly,
but performing this with standard proto
cols like NTP (Network
time Protocol) can be tricky.

Although the bucket synchronisation method is not directly
suitable for fast
paced FPS games due to delayed event
processing, MiMaze compensated for this latency by predicting
the next state using the la
st valid message

as our proposed
system should do. However, this system was directly embedded in
the game
logic and not a reusable system as we are trying to



We have identified components which are necessary for setting up
a state synchronisation system between multiple networked
computers. Our plan is to build the system in smaller stages

time in order to make the project more manageable.

Firstly, the basis for the game state management is required,
which is made up of both data storage and event storage. This is
required to have state interpolation and extrapolation features
which a
re then used in state prediction in order to save bandwidth.

We found that a good solution is to use a database that supports
transaction time support, allowing the system to obtain previous
table states from the database. Although this level of support is

not present in many databases due to the overhead of storing
previous state information, we have found some systems that
allow this including Immortal DB

and PostgreSQL with the
travel pl


through the SPI (
Server Programming
). It appears to be useful to have this support in the
database engine rather than Java, as storin
g this information using
Java data structures would create a large footprint due to the
heavy use of objects
. Although some projects such as Project
Darkstar have opted to implement their own Jav
based state
database which allows retrieval of previous states, but can also
use other types of database if desired, including those that are

With this database, we want a simple me
thod of linking up state
variables with database data, so we can say that there is a standard
variable indexing system which maps into the database and must
be defined in a configuration file. Although in the long run this
should optimise multiple queries
into a single, joined query, this is
not essential in the short term. The system should handle the
filling of data into the object interface, but should also

Alongside this previous state information, we can also store event
information which is relevant
to both the game state and
synchronisation system state. This information includes details
such as client connections, disconnections, synchronisation
events, game events, etc. This event storage system comes in
useful, as some state details are difficult
to store in the same
fashion as the state data. It may be possible to use this event
information alone to obtain previous states, but processing the
event information to achieve this may be expensive. However,
this remains a strong possibility in the futur
e construction of the

We also require the node management system which will oversee
the construction and maintenance of the hierarchical topology of
the network, and will control the operation of any type of
linked server operating on these

nodes. The system will
operate on a different socket with a confirmed protocol to the state
synchronisation system, and will inform the running system that a
level stable link has been created between two network
nodes. This system will be updated wit
h the status of the
connection and management during the lifetime of the service
through a listener interface.

Essentially, this system runs the state synchronisation system on
each node and acts as a failsafe for any problems. It will be up to
the state m
anagement system to set up its own high performance
link based on the information received from the underlying
system, which will use a different, high
performance protocol
such as RTP (Real
time Transport Protocol) to transfer frequently
changing state da
ta. Additionally, the state synchronisation system


Although PostgreSQL had this support in earlier versions (pre
), it was removed and converted to a plug
in for
mance reasons

itself may use a lower performance, high assurance protocol such
as TCP/IP to synchronise data less frequently.

This node management system will run as an OSGi service, as this
provides lifecycle managemen
t and should allow the system to be
used in various OSGi service managers.

For the initial simulation, we would also like to use Eclipse IDE
to display the state of the network as it is running. For this, we
should be able to use standard JFace components
in standard
workbench views and manipulate them as the simulation



The type of topology that the system will be designed for is a
hierarchical, node
based, non
recursive network. The network is
designed to act ‘behind the scenes’ to spr
ead game state data
efficiently while allowing the client to connect to particular nodes
and obtain and modify the state data that is relevant to themselves
only, in their area of interest.

Usually, the developer would have to create and manage this
gy themselves through their own logic, and then manage
bandwidth through careful event handling and propagation. The
idea is to make this generic, usable in any kind of networked
system without tying the implementation to the idea.



The config
uration files will most likely use XML to allow the
system to be configured and to set up links into the game logic.
The network configuration will define the slots and the servers
that fill the slots in the overall hierarchy. The synchronisation
system co
nfiguration will define the data types within the system
and the specifics of the range of variables that use these types. It
will also describe how the defined data types map to the database,
and the also how they map to the Java object interface. A tool
could be produced which would generate the necessary Java
interface classes.



The system will eventually optimise the state data transfer using
the natural hierarchical topology of the system to provide area of
management. Although this is done

in principle in many existing
games, it is generally manually implemented without an
underlying framework at the expense of time. Also, it is difficult
to implement the features in a game that this framework will
eventually provide.

As an example.
a M
G such as World of Warcraft™ has an
organisation where

physical network layers are oriented with the
different geographical regions of the game itself. For instance,
there are multiple realms containing a world with continents,
zones, and regions, with ins
tances, all of which are mapped to
ilar server network structures.

This is done to consolidate server data which improves data
security and reduces downtime, but also to reduce processing
requirements by spreading the game world across multiple
s and allowing distributed computation. Also, it allows
more clients to be connected to the game at any one time by
providing multiple client connection endpoints, and reducing load
by not propagating certain information upstream. For example,
movement inf
ormation may not be propagated to the world or
realm servers, even though these servers may be aware of what
realm and zone a player is in for game management purposes.

: World of Warcraft

game realm selection

ly, it allows particular configurations of game across
different realms, where one server may be PvP (player vs. player)
where players can attack each
other or PvE (player vs.
environment) where the focus is on quest completion and
defeating monsters to le
vel up your in
game character

: Example of MMORPG Hierarchical Topology

We can say that there are two types of knowledge for state data

knowledge and full knowledge. Full knowledge is good
for a short amount of time, as we can base our derived knowledge
off the state from it. However, we do not always have full
knowledge as sending streams of data to keep state variables
always updated wou
ld overload the system

this is precisely the
problem with many state synchronisation systems.

In order to extrapolate the future state in our system, we can use
this event information alongside the previous state data values. In
an ideal system, key even
ts should be taken into account when
generating this information, but storing this information would be
too expensive, and would make prediction computationally

To extrapolate a value from the previous state, this is far less
computationally i
nvolved and requires the surrounding values in
the timeline. For this, key event retrieval is required to indicate
where certain values become unpredictable from regular
interpolation. It is possible to imagine a situation where the user
moves and then die
s, where the event store would describe this
death event with relative information, but the state storage would
merely relay the position of the player.

Performing interpolation between a value before death and a value
after death, ignoring the death event
, would give values that bear
less relation to the true values at that time than if the key event
had been taken into account. The state information is needed to
inform the system where the player was at death and at respawn
so that the correct state data
can be derived; the values before
death can be ignored.

There are two components to each distinct state member when
there is a higher authority acting upon the state, which
traditionally both exist in distinct store spaces: the real state, and
the requeste
d state. The requested state is generally updated with
user input and is used by a state authority that also acts as a state
transformer to transform the real state to match this requested
state. For instance, the user may request that their character move

to a certain position at regular intervals. The client will
extrapolate of this state, even before confirmation from the server
they are connected to. Alternatively, future intended positions
may not be stored, but can be retrieved using the time, charact
velocity and orientation. In this case, a decoding step can be used
to write the missing intended spatial variables.

One of the problems of this method is that there are often other
contributing factors that will change the prediction. For instance,

logic on both the client and server will enforce that a character
cannot travel through the scenery or terrain (although it is the
server logic that enforces this by controlling the real game state).
In fact, it is the game logic which dictates how state
will be extrapolated, so there needs to be some form of cross
communication between the game logic and prediction engine.
Indeed, the prediction engine may need to lie inside the game
logic for the most part, using utility functions from the
nchronisation system when needed.

Hidden information that a node is not privy to (perhaps this is
because of security reasons) will also affect the predictions
between nodes. For example, there may be a hidden player that
one node is not aware of, but anot
her is not. However, for now the
system should assume that both nodes will predict the same
information by providing the hidden information. We can feed
this information into the predictor, but hide it from the client.

: Example of MMORPG hierarchical topology



Messages are required to keep the system operational and provide
a standard basis for communication. Messages can have assured
sion or non
assured transmission. We can also timestamp
each message to help with algorithms later on, and also to make
sure that nodes are aware of when a message was generated. We
should also timestamp message contents, like events.

Important messages
that are required to set up and maintain the
state synchronisation network include server network connection
requests, acknowledgements and disconnection notifications.
Also, messages should feed through the network which inform of
node hierarchy changes.
Generally, we need to keep this hierarchy
static, so the slot system simply informs the network of what the
slots are, who is filling the slots, keeps the network updated with
this information. It should be okay for the master server to handle
the state of

the network, since the amount of data should be

The client does not need to know about the topology of the
network, only the IP of the zone they will connect to. Clients
should first communicate with the server they believe they will
w them to login to check if a connection should be established.
If the master server demands that the client connects to another
server, this should be negotiated (e.g., the game is reset and player
world positions change).



Events are a particular t
ype of message which indicate state
transitions, and are often linked to user action. Different events
will require different state data to be synchronised (a partial
synchronisation), which is important for future predictions and
interactivity maintenance
. We can define these dependencies in
XML using a simple system of grouping.

We can say that some variables, such as movement or rotation,
rely on the spatiality of the player character. Therefore, we can
group these variables into categories that describ
e different
aspects of the state. In this case, they would be grouped into a
‘spatial’ category and we would use this to inform the system that
some variables in this group must be synchronised at that point in
time. As each node knows how variables are gr
ouped, a
synchronisation call can be reduced down to its base necessities to
reduce bandwidth.

As mentioned earlier, events may also be useful in backtracking to
previous states, although there may be a computational overhead
associated with this.



The prediction engine feeds values into the game model as it
predicts them, in order to ensure that the level of interactivity with
the game is kept as high as possible. Ideally, all players should
have the same game state such that

for any pair
of players

and at any time instant
, but this is not possible
in networked environments due to network latency. With

being the network delay from player

to player
, a
n action
performed by player j at time

will become known to

at time

. Instead, we need to
State modification request

Internal processing

External processing / input


Local state


extrapolate future information from past information,
error in the state when new information is fed from other nodes.

We need to monitor state inconsistencies, and correct the
inconsistency or synchronise the state when these are detected. A
particular inconsistency might be the collection of a we
apon by a
player character in an FPS game when the character is not yet in
that spot, although feeding this information back onto the game
model should force partial state synchronisation of dependent

Using technology both researched and in use
will also help. There
are currently some systems which make state synchronisation
more efficient and usable, but each with their own advantages and
disadvantages: the time bucket system which MiMaze uses and
which was discussed earlier, and the time warp s

is based on the use of state snapshots and rollbacks.

Many current synchronisation mechanisms in use were designed
initially for usage in large
scale military simulations, and the
oncepts were only later used in multiplayer games as computer
gaming gained in popularity

The main disadvantage with the time warp system is that state
snapshots are required to allow roll
ack to previous states if an
earlier event is received, so that the system can re
execute events
up to the current execution time. The problem here is the state
storage, known as the ‘execution context’, which can become very

Also, managing data str
uctures in the execution context such as
linked lists can also be tricky. It may be possible to optimise this
system by allowing certain events out of order, or ignoring an
event and managing the problems that result. We could also buffer
certain events in
dependent of other events, and could then use an
event dependency system to find out which event executions are
dependent on the execution of other events (i.e., a speech event
does not directly affect movement events, but out
order delays
may slightly
affect interactivity.)

Ferretti and

et al.

propose the use of an indicator known as
the Game Interactivity Threshold (GIT) to calculate the maximum
endurable time before an event must be displayed on the screen of
the client to maintain the semblan
ce of interactivity
This can then be used as a factor in working out when to predict
values and when to synchronise certain elem
ents of the state.

The formula for calculating the
GIT is
Here, we can see GIT which is a product of the overall latency
(OL), which is in turn made up of the network traversal latency
(NTL) and the last hop latency (LHL), such
. NTL refers to the game event
transmission time, from event generation to reception on the target
machine, and
LHL refers to the time it takes to process the event
once received on the target machine

: Diagram of event delays

For each server, certain information should be stored ab
directly connected servers (direct peers) to optimise
synchronisation and hold back assured information. Firstly, it
would be prudent to know if any changes have occurred to a
variable since the last synchronisation to avoid retransmission in
the next
partial or full synchronisation. A full forced
synchronisation would transfer these values regardless, which is
useful when the state may be corrupt or missing, such as when a
client or server initially connects to the network.

Additionally, we need to mon
itor when the margin for error of
certain game variables become more than is tolerable. In short,
this is when the difference of the extrapolated value and the real
value become greater than is defined as a percentage value of a
scale. In order to do this,

nodes may need to store partial states of
other directly
connected nodes, both sent and predicted.



The game should have its own set of classes which use the state
management interface classes. Although this seems like a more
complex solut
ion with more resource overhead, this is important
as the client game view and model provides an element of
dynamic interpolation which the state synchronisation system
cannot provide. For instance, the system may say that a player’s
position has changed,
but snapping to the new position may cause
a loss of interactivity. The input from the system is a stream and
should be interpreted as such. Also, it should not be the case that
the entire game manipulation logic is in the state synchronisation

is would be a far too complex approach which would
not allow the mixture of model and view that is commonly used in
games to simplify the game code.

In the model, we can use predictor logic stubs to allow the user to
define their own state variable extrapo
lators and interpolators
which are more complex than the generic ones. These stubs
should be defined as classes that do not require multiple instances,
and the relevant classes could be referenced in configuration files,
and instantiated using reflection.
In these stubs, we can pass or
make accessible the required data for calculation in methods that
should be overridden by the developer to provide functionality.



In this paper, we reviewed our work so far on the Koku project.
Although there are o
ther established state synchronisation systems
out there, we feel that there is room for improvement and we aim
to look into this through further development of our system. We
particularly need to work on use
cases involving new games and

Game event sent between nodes



Sending GSS

Receiving GSS

OL (overall latency)

NTL (network traversal latency)

LHL (last hop latency)

existing games, s
tepping through how the system will work at
each point, and how the state will actually be managed.




T. Hsiao and S. Yuan, “Practical Middleware for
Massively Multiplayer Online Games,”
IEEE Internet
, vol. 9, 20
05, pp. 47


J. Waldo, “Concurrency, Distribution, and Server Scaling,”
Feb. 2008;


C. Diot and L. Gautier, “A distributed architecture for
multiplayer interactive applications on th
e internet,”
Networks magazine
, vol. 13, 1999, pp. 6


L. Gautier and C. Diot, “MiMaze



L. Gautier, C. Diot, and J. Kurose, “End
transmission control mechanisms for multiparty interactive
applications on the Internet,”
INFOCOM '99. Eighteenth
Annual Joint Conference of the IEEE Computer and
Communications Societies. Proceedings. IEEE
1999, pp.
1479 vol.3.


D. Lomet et al., “Transaction Time Support Inside a
Database Engine,”
Proceedings of the 22nd International
Conference on Data Engineering
, IEEE Computer Society,
2006, p. 35;


“PostgreSQL: Documentation: Manuals: PostgreSQL 8.3:
spi (F.27.2. timetravel.c)”;


J. Shirazi, “Java Performance Tuning,” O'Reilly &
Associates, Inc., 2002, pp. 94


“Sun Labs

Project Darkstar: How it works”;


T. Surette, “GameSpot: WOW patched to v1.4”;


J.D. Pellegrino and C. Dovrolis, “Bandwidth requirement
and state consistency in three multiplayer game
Proceedings of the 2nd workshop on
Network and system support
for games
, Redwood City,
California: ACM, 2003, pp. 52


M. Damitio et al., “Comparing the Breathing Time Buckets
Algorithm and the Time Warp Operating System on a
Transputer Architec
Proc. of the European
Simulation MultiConf.(Barcelona, Spain, 1
3 June 1994)
pp. 141


E. Cronin et al., “An Efficient Synchronization Mechanism
for Mirrored Game Architectures,”
Multimedia Tools and Applications
, vol. 23,
2002, pp. 7


S. Ferretti et al., “Buscar el levante por el poniente: in
search of fairness through interactivity in massively
multiplayer online games,”
Consumer Communications
and Networking Conference, 2006. CCNC 2006. 3rd IEEE
2006, pp. 1183


C. Palazzi et al., “On maintaining interactivity in event
delivery synchronization for mirrored game architectures,”
Global Telecommunications Conference Workshops, 2004.
GlobeCom Workshops 2004. IEEE
, 2004, pp. 157


“PostgreSQL: Documentation: Manuals: PostgreSQL 6.3:
Time Travel”;