Tweaking Multiplayer

[Immortius]

Name: Multiplayer Arc
Summary: Implement Multiplayer Support
Scope: Engine
Current Goal: Entity Serialization cleanup
Phase: Implementation
Curator: Immortius


I realise I have a number of other Arcs with outstanding tasks (primarily modding), but getting multiplayer implemented is important as it will impact on how other systems will work. I believe things have reached the point where multiplayer can be added so this will be my primary focus. Once multiplayer is done I'll return focus back to modding, cleanup, documentation and some actual gameplay.

Still need to work out the details, but what I envision is involved:

• Basic connection support (Host listen server, join server).
• Chat support.
• RemoteChunkProvider and chunk transfer support.
• Proper connection handshake with configuration transfer (active mods, block to id allocation, etc)
• Chunk updates, various other world related updates (particularly notification when a block is allocated to an id due to lazy allocation)
• Network Component
• Entity replication (probably involves annotations on component fields)
• Event replication
• Player persistence between sessions
• Headless server (Dedicated server)

Out of scope for this arc:

• Master server, identity server or any sort of server browser.
• Rigorous cheat protection/prevention
• Automatic download of needed mods (too dangerous without mod sandboxing)
• Probably Source engine style client-side prediction of character movement. Assuming we want it.

My intended technology stack is Netty using TCP for this push.

Work Complete:

• Basic network system and connection establishment
• Bits of the connection handshake - server information flows to client during loading
• RemoteChunkProvider
• Transfer of chunks and block updates
• Basic entity replication
• Event replication
• Simple chat
• Basic Client/ClientInfo/Character setup
• Inventory replication
• Inventory behaviour and world interaction netcode
• Improved control over the replication of fields
• Inform client when blocks are allocated ids after joining
• Improve handling of block update messages
• Remove lighting information from chunk messages, and have the client recalculate lighting
• Proper connection failure/disconnect behaviour

Outstanding work:

• Optimisations, generally via one-off replicating lookup tables (asset uris -> ids) and flow control
• Improve chat interface

 

[Immortius]

Starting to get some results with multiplayer now. There is still a lot to go, but thought I'ld quickly go over some features that relate to development and modding.

The three cornerstones of the network implementation are world replication, entity replication and event replication.

Network Mode

Currently there are three network modes: SERVER, CLIENT or NONE. SERVER and NONE are authoritative modes - the player's game controls everything, the world is in their control. This information is important for systems to make determine what they do on a given machine - and whether those systems should exist on those machines.

A further dimension is whether the game is headless or not - does it render and have audio? A headless server is effectively a dedicated server. Some systems won't be needed on headless machines (anything that renders), and various rendering, audio and input code won't run. Some things related to those will still be needed (if the animation of a mesh is gameplay critical, the mesh will need to be animated on the server even if it isn't rendered).

World Replication

At the moment, any changes made to the world through the WorldProvider on the server are automatically sent to all clients, where they are reapplied. This is pretty much makes world replication something that doesn't need to be worried about much.

On clients the world can still be changed, but these changes are not replicated. It is intended that this can be used for hiding latency (by predicting changes).

Entity Replication

There are two parts to this. Firstly, if an entity needs to be replicated, then it needs to have a NetworkComponent. This network component holds its network id - unlike entity ids, the network id is the same on all machines. I decided not to try to keep entity ids the same across all machines as being able to create client-side entities freely felt important (e.g. you might want to create a particle effect client-side only).

The NetworkComponent also holds the ReplicateMode of the entity. These are ALWAYS, RELEVANT and OWNER. ALWAYS means the entity will be replicated to all players, wherever they are. RELEVANT isn't implemented yet, but will mean entities will be replicated to players in a certain range. OWNER means the entity is only replicated to it's owning player - no one else needs to know about it. This could possibly do with some extension - for a chest, the items in the chest should only be know to players with it open. So a concept of multiple owners or viewers may be better. A chain of control would also be good, so that the items can be owned by the chest, and then the chest is owned by one or more viewing players. At the moment ownership has to be direct.

Once the entity itself is set to be replicated, what is currently replicated is what components it has, and any field in those components that is annotated with @Replicate(). e.g.

public class DisplayInformationComponent implements Component {
    @Replicate
    public String name;
    @Replicate
    public String description;


    public String toString() {
        return String.format("DisplayInformation(name = '%s', description = '%s')", name, description);
    }
}

Still to be considered
* Marking which components are replicated
* Having some fields replicate from client->server
* Improvements to the entity system to allow change deltas to be replicated instead of the entire entity
Definitely Needed:
* Conditional replication. It is necessary that certain fields depend on some sort of condition to determine when they will be replicated. For instance, location's position and replication should only be replicated if they are not under physics control, and are not controlled by a character controller (which will use a separate system). Perhaps having boolean fields that can control this?
* Tear-away entities. This are replicated when created, but then immediately broken off to act independently on each machine with no further replication. This is good for server-driven particle effects for instance.

EntityRefs over the network

Worth mentioning is that if you receive an EntityRef over the network (either in a component, or as part of an event), a special implementation is used that acts as a null entity if the entity has not been replicated, and as the actual entity when it has been. This happens automatically, and can change frame to frame as the entity become relevant or irrelevant.

Event Replication

Event replication is a little simpler. Once again only fields marked with @Replicated will be sent. If an event is annotated with @ServerEvent or @OwnerEvent, then it will be replicated to server or the entity's owner respectively. If the entity extends NetworkEvent it has an additional field for the client it was received from on the server.

Replicated events also take place on the machine they are sent from, to allow any desired predictive behaviour to occur.

 

[Immortius]

Progress continues. Player characters now are created and replicated to players. The set up is that there are 3 main entities involved with players - a Client entity, which represents the player's connection (and thus doesn't persist), a ClientInfo entity with details on the player that are shared globally, and a Character entity which is the player's presence in the world. With the player's character now created and linked to them, they can move about and new chunks are streamed to them as they travel.

The location of the player's character is replicated from the client to the server now, using new support for replication from owners back to the server. This is far from ideal as players will jerk about using this mechanism (and could teleport-hack), but it provides a simple beginning.

I also have the player's inventory replicated to them now, although interaction with the items is not in yet. This is demonstrates the ownership chain - they items are owned by the player's character entity, which is in turn owned by the player's client entity - and the items are only replicated to that player.

A final thing I've added are metrics on incoming and outgoing bandwidth usage.

 

[Immortius]

Another update. At this stage clients can now use items in their inventory and attack - which means they can interact with the world a little. The implementation is crude, and will need to be cleaned up when client movement is updated to use a client-predictive model.

Before that I want to optimise the amount of data being sent around. To give some metrics, having a single client logged into the server resulted in about 20k bytes being sent to the client each second, and 4k bytes being received from the client. This data? The client sending an update of its rotation and location 20 times a second, and the server pushing full updates on both its and the clients entity 20 times every second.

Ignoring the fact that the this is not the intended replication model for client movement, it gives a decent test case for seeing what can be improved with the current implementation. There are two ways I'm tackling this:

Improve Message Format

The first thing I did was change the serialization of entities so that instead of using strings for the names of components and fields, using id numbers. These mapping of names to ids are propagated to clients when they connect. This changed the sent data to 12k a second, and received to 2k - nearly halving it. There are a couple of other things that could be replaced with id tables (enums and asset uris) as well.

I'm also having a look at the structure of messages- by flattening the structure to make better use of repeating elements, I'm hoping to get a further reduction in data size.

Improve Entity Change handling and Update Messages

At the moment, when an entity changes (any component saved, component added, component removed), it is marked as dirty. When it is replicated, a full list of all of its components, and their replicated fields are sent. This is way too much information, and way too easy for entities to get marked as dirty.

Ideally what we want to send is

• Components added to an entity, but only the ones that need to be replicated, as a delta off their default state.
• Components removed from an entity if they were originally replicated
• When a replicated field of a component changes, it should be replicated - but only the fields that have changed (so the delta is sent)

So work to be done is:

• Improve the concept of a replicated component, and the handling around their adding + removing
• Track dirtiness at the component level, so only changing components are replicated
• Having some fields only replicated initially (so not considered for updates)
• When a component changes have an original state to calculate the delta off of

This third point requires some changes to the way the entity system works, but is doable.

 

[Immortius]

Another week. Implemented many of the improvements above, resulting in bandwidth of 2.3k out, 1.8k in a second. Could be even better, but this is decent enough to continue.

Started working towards a proper clientside prediction model for character movement. Have changed the movement system to be driven by CharacterInputEvents and CharacterStates, with an input + an original character state producing the next state. These can both be stored in buffers, which will allow the sort of time-travel shenanigans needed for the system.

One upshot of this is directly changing character locations and movement state (like velocity) no longer works - they are overwritten by the state held by the CharacterMovementSystem. There will need to be a couple of new events for teleporting the player and switching in and out of ghost mode.

Next is getting client side behaviour correct for the local player - they should be sending their input to the server while retaining it locally to generate predicted states, and then later when actual state resulting from the input is received dropping the input and updating the predicted state, with either a full correction or a partial correction depending on the scale of any deviation.

After this is remote client behaviour - how clients deal with the state of other players/creatures. This will require interpolation and extrapolation of character state, and a delayed playback of 50-100ms (to allow some state to be buffered for interpolation). Interpolation is necessary because state will only be received a limited number of times per second, likely less than the players framerate.

Finally interactions will need to be updated to account for latency - characters will need to be rewound to previous state for use item, frob and attack, primarily for hitscan interactions.

 

[Immortius]

Character movement is now using a server-authoritative, client-predictive system as above. When a player uses an item or attacks, characters are rewound to their positions at the time of the user's action (theoretically to allow interaction, although attacking does nothing yet). Some tweaking is still required but it feels solid.

With this all the fundamental building blocks are in place. I'm now starting to work my way through the core gameplay mechanics, updating them for multiplayer. I'm starting with removing blocks and placing blocks, which will touch on inventories as well. As I go I'm tweaking the network API and fixing any issue I come across, which has already uncovered some trickiness with broadcasting events against blocks or newly created entities.

A quick peek at one of the advanced networking features - if fine grained control is required over the replication of fields in a component, then a component can implement the ReplicationCheck interface and shouldReplicate() method. For instance, if SomeComponent's angerRating should only be replicated if it has a target, then you could do something like:

public final class SomeComponent implements Component, ReplicationCheck {
    @Replicate
    private String name = "Blah";
    @Replicate
    private EntityRef target = EntityRef.NULL;
    @Replicate
    private float angerRating = 0;
 
    @Override
    public boolean shouldReplicate(FieldMetadata field, boolean initial, boolean toOwner) {
        if (field.getName().equals("angerRating") {
            return initial || target.exists();
        }
        return true;
    }
}

 

[Cervator]

Had to think about it for just a bit, but it makes sense. shouldReplicate gets queried by the replication system, includes whether or not it is the initial replication, in which case initial wins over whether or not the target exists (a special case in this example)

 

[Panserbjoern]

Hi Immortius. I guess, support for stackable chunks will have to be added manually to your multiplayer arc. Or do you plan to merge with the singleplayer branch later?

 

[Immortius]

I think I've resolve to handle multiplayer as a fork, and manually merge select changes from singleplayer later, yeah. It isn't great, but I can't really think of a way around it.

 

[Cervator]

We can strip out most the modules and probably handle them separately fairly easily, but yeah, engine stuff might be rough for a bit

 

[Panserbjoern]

Well, i'll do the chunk stuff, if that helps.

 

[Immortius]

I've already integrated the existing chunk work with your new representations and serialization (I grabbed it as soon as I could ).

 

[Panserbjoern]

Cool! But stackable chunks will introduce lots of changes. Looooooots... Might lead to some collisions.

 

[Immortius]

No offence, but I think you're overengineering that change.

 

[Panserbjoern]

None taken! I know, i tend to overengineering when i have too much time... I promise to keep it as simple as possible.

 

[Immortius]

Progress report: Breaking blocks, picking up dropped blocks and placing blocks are all working in multiplayer now. I plan further tweaks to the way this works (mainly keeping a concept of "held item" on the server, which could be propagated to clients to render it attached to each player's hands). I may also bring the actions in line with Overdhose's suggestion too (merge attack and use item).

I did a bit of an overhaul to the audio system along the way. There are now events that can be used to propagate sounds - PlaySoundEvent and PlayOwnedSound event. PlaySoundEvent is sent against the entity making the noise, and broadcast to all players for which it is relevant, optionally excluding the instigator of the event (if the sound is simulated for that locally for that client). PlayOwnedSound is propagated only to the owner of the entity. In adding this I discovered that positional sounds were broken (they because inaudible at 3 blocks distance), so I fixed this and did some overall improvements to the system's API. This includes support for music and sfx volume control, which is important for testing that sounds are playing properly in multiplayer (I turn off sound for the server or the client to test the other).

Next I'm looking at the inventory system. For this I'm going to experiment with having a manager class rather that relying on events for everything - so rather than different systems accessing the inventory component directly or sending a ReceiveItemEvent and hoping things work out, they will get an InventoryManager or SlotBasedInventoryManager out of the CoreRegistry and use the methods in this. InventoryManager looks like:

public interface InventoryManager {
 
    /**
    *
    * @param inventoryEntity
    * @param item
    * @return Whether the given item can be added to the inventory
    */
    boolean canAddToInventory(EntityRef inventoryEntity, EntityRef item);
 
    /**
    * @param inventoryEntity
    * @param item
    * @return Whether the item was fully consumed in being added to the inventory
    */
    boolean addItem(EntityRef inventoryEntity, EntityRef item);
 
    /**
    * Removes an item from the inventory (but doesn't destroy it)
    * @param inventoryEntity
    * @param item
    */
    void removeItem(EntityRef inventoryEntity, EntityRef item);
 
}

and SlotBasedInventoryManager extends this with some methods involving actual slots. The actual implementation is a standard ComponentSystem, registered with the new @Share annotation:

@RegisterSystem(RegisterMode.AUTHORITY)
@Share({InventoryManager.class, SlotBasedInventoryManager.class})
public class InventorySystem implements ComponentSystem, SlotBasedInventoryManager {

The Share annotation causes it to be registered into the CoreRegistry, which in turn allows it to be injected into other systems using the @In annotation.

@In
private InventoryManager inventoryManager;

The theory is that gametypes will be able to replace the inventory manager (and things like it) with their own implementations that will still work with existing code (because the existing code works against the interfaces), but provide behaviour the gametype needs. Mods could potentially do this too, but if multiple mods change the same system only one will succeed. If there are some obvious types of modifications those can be explicitly supported though.

 

[Immortius]

Things changed a bit, but inventories and chests now work in multiplayer. Took a couple of iterations to get it working nicely and had tear into the inventory GUI pretty heavily as well but it is now set up with a predictive model. When you move an item its new position is predicted while the request to make the move is sent to the server, and at some point in the future the server sends back an acknowledgement with the new state of the inventory. At this point the prediction is dropped and any further changes are recalculated off of the new state. This gives a very strong illusion of immediacy on the client, with the rare undoing of an action if another player grabbed the same item out of the chest or the prediction otherwise failed.

Further from this I've been working on cleaning up some rough edges:

• A number of bugs in the network handling, mostly around block entity handling, have been fixed.
• A failure to connect now results in an error message rather than a crash.
• When a client drops, their character is removed from the game, as is the server player's character when they shut it down.
• When the server shuts down or a client is otherwise loses connection to the server, they are sent back to the main menu with an error message
• Clients now determine their own view distance independent of the server player, and can change it during play with the server dishing out the correct chunks.
• Cleaned up the join menu a little, and fixed some UI issues resulting from changing all UI elements to be visible by default.
• Incorporated the configuration handling changes from develop.

 

[Cervator]

Always good to hear updates

Let us know when you need some additional testers

 

[Immortius]

I've been focusing on getting some of the chunk handling sorted out, particularly around flow control. To explain a bit, lets say we want a 512kbps upstream connection to support 4 remote players to start with (that is a fairly small number of players, but if gametypes are more focused on having a few friends rather than large communities then it is alright).

That means that the server has a total of 64 KB it can send each second, or ~16,000 bytes per player. At the moment I've set it so that a maximum of 7 chunks are sent per second to each player, which falls within this range. These chunks are also spread among the communications that occur over the second, so as to prevent the remote players being starved of entity updates and events.

I have also stopped sending lighting information along with the chunks - it is now up to the clients to recalculate lighting, which works due to lighting being deterministic based on blocks alone.

Finally block id allocation is now replicated to remote players.

 

[Immortius]

More work on the chunk handling. Block updates is now properly supported - I made a variety of tweaks to the chunk generation pipeline and chunks themselves along the way. Renamed WorldView to ChunkView and tightened up on chunk view generation.

Server upstream bandwidth is now configurable, and is dynamically divided among connected players.

Started doing some more work around players again. Footstep and jump/landing noises are working again, as well as fall damage. Death isn't properly implemented yet, but that will be next.

To give some idea where I'm at with multiplayer, this is some of the items remaining on my TODO list in no particular order. It segues into engine cleanup and mod support a little:

• Death
• Respawning
• Dropping items
• Force inventory closing when moved away from chest
• Add Inventory content restrictions (e.g. bookshelf)
• Destroy inventory transfer entity and eject contents on character death
• Eject inventory contents on character death
• Simplify item usage
• Improve owner support
• Entity Relevance (as pertains to replication)
• Maintain selected item on server (for future use rendering held item)
• Refactor inventory/items into core module
• Fix event receive when target is missing
• Player persistence
• Improve chat
• Chat filtering
• Overhaul command system (make all systems eligible for commands, fix up commands for multiplayer)
• Turn chat into a command
• Connected players list
• Copy components on get (sanely)
• Change player character spawning to occur after area is loaded on the client side.
• Improve character prediction (deal with initial lag, deal with large scale desynchs)
• Improve physics synchronisation
• Improve handling of times (game time, world time, timers)
• Dedicated server