Persistance

[Kai Kratz]

I like to discuss a little about persistance with you guys.

The tasks at hand:
Begla would like to see binary and human readable(XML) serialization.

What i would like to do (or rather recommend since I have enough tasks already)
a) Custom serialization for storage heave components (our world files - nothing else)
b) Java serializable for binary serialization i.e. release builds
c) XStream serialization for debug builds

If I interpret the serialization code of immortus correctly hes working on point a)
Really would advise however against doing more than absolutely necessarily custom serialization. Resolution of dependency graphs is time consuming to implement properly (done it once partly before switching to boost)

Please share your insights would very much like to hear your opinions on this

 

[Immortius]

XStream looks nice, I especially appreciate the JSON format support (I find it easier to read and a little less verbose).

With the Binary Persistance thing I did, it was originally written to work with an entity system. In an entity system you only have Entities, Components and the data in the components. Entities are just IDs, you don't even have explicitly store them - you create them as they are referenced. Components are owned by a single entity, and can also reference other entities, but never each other (or if they did it would be by entity + component type, as entities only could have a single component of each type in the model I was using). Finally the component data was reasonably simple stuff, not complex object structures, and some of it can use a flyweight pattern to reduce instances. So there wasn't any complex dependencies to deal with in that case.

Certainly I agree that if we are serializing anything with complex dependency graphs to use something else.

The other option I have been considering in my experiments with Entity Systems is some sort of database - atomic transaction support and optimistic version control would provide the basis for a highly threaded, redo-on-conflict style server. But that is mere theorycrafting at the moment.

 

[Cervator]

Moved to Dev forum

Also need to consider the impact of the choices on future networking potential

What's with b) release builds? Eh?

And XML? Eeee

I saw a Groovy prop writer begla did a little ways back for simple stuff. I know the config serialization to plain text didn't go too well with a more layered setup, but do we need much more than that in plain human-readable format?

On binary persistence - I'll let you all sort that out

 

[Kai Kratz]

Moved to Dev forum

Also need to consider the impact of the choices on future networking potential

What's with b) release builds? Eh?

And XML? Eeee

I saw a Groovy prop writer begla did a little ways back for simple stuff. I know the config serialization to plain text didn't go too well with a more layered setup, but do we need much more than that in plain human-readable format?

On binary persistence - I'll let you all sort that out

There are several issues with the ConfigSlurping of groovy but ill skip that and mention only one issue which makes this feature unusable in my opinion.

ConfigSlurper + ConfigObejct can not do a round trip when writing config files (only with primitive types and strings)
ConfigSlurper can not parse the files generated by writing out ConfigObjects if they contain class types. Serializing ConfigObjects trough their write method will garble any class type. So why use groovy when you cant use the part that made you use groovy in the first place? Really like the groovy approach but sad to see it implemented so sloppily.

 

[Cervator]

Yeah, I agree. There are some user contributions that seem to help the issue, like that one that allows "imports" in these super prop files of other super prop files. I wonder if somebody has made the round trip issue work somewhere?

Edit: Adding links to a couple existing issues - https://github.com/MovingBlocks/Terasology/issues/157 (config files) + https://github.com/MovingBlocks/Terasology/issues/139 (persisting per-block data)

 

[Kai Kratz]

I did toy around with Google Protocol Buffers aka. Protobuf and I really like it. I will build a little prototype and push it up to my fork. Think ill be done with it Friday or Saturday.

 

[Cervator]

Looks very interesting

 

[t3hk0d3]

IMHO we need 3 serialization ways:

1) rigid and compact (serializing specific data), for networking or storing chunk data (protobuf)
2) to serialize generic data into binary format - sending generic objects over network (any suggestions?)
3) human-readable serialization (gson?) - for human-editable tasks (configs, modding, prefabs, etc)

We should make agreement on that to avoid serialization zoo.

 

[Cervator]

Pulled the initial stuff into Develop. Since Kai and the applet were both involved I of course found something to fix

https://github.com/MovingBlocks/Terasol ... ed1e19457e

I'll give partial credit tho, it didn't break the applet! It just didn't work for the applet

Also, only the view distance seems to persist. Graphic quality seems to revert to ugly?

 

[Kai Kratz]

Pulled the initial stuff into Develop. Since Kai and the applet were both involved I of course found something to fix

https://github.com/MovingBlocks/Terasol ... ed1e19457e

I'll give partial credit tho, it didn't break the applet! It just didn't work for the applet

Also, only the view distance seems to persist. Graphic quality seems to revert to ugly?

Look into it tomorrow (or rather later today)

 

[ironchefpython]

IMHO we need 3 serialization ways:

1) rigid and compact (serializing specific data), for networking or storing chunk data (protobuf)
2) to serialize generic data into binary format - sending generic objects over network (any suggestions?)
3) human-readable serialization (gson?) - for human-editable tasks (configs, modding, prefabs, etc)

We should make agreement on that to avoid serialization zoo.

I disagree. You need one serialization mechanism. This one mechanism needs to support all types of information associated with a world, from the raw block data to the metadata added by arbitrary mods. It needs to support representing subsets (chunks of the world) and should support changesets (change since a prior state). It should be compact, without requiring CPU heavy compression/deduplication/graph reconstruction/etc.

It must have an associated API that can be scripted. This API will allow things like modification of save games, network debugging, etc. It will also allow you to run a script to dump the world or portions of the world in a human readable format.

As far as protocol buffers go, it's a nice little tool when language independence is required, but it requires a compilation step to change the message structure. So in order to extensibly store arbitrary data (such as that added by mods), you'll need to add a repeated Key/Value type in your messages.

 

[Cervator]

I think (may be off) that the serialization note there relates to how we need to both serialize stuff for network transport (binary) as well as serialize stuff like simple settings to disk, which would be stored in plaintext

To a point the world manifest files may also relate here. Currently we save mappings between block names and the images used for their assorted faces and such in plaintext. Block textures are merged together into one big image, then saved in a few different sizes, stuff like that. Can actually edit what textures are in use in a specific world via notepad

And thanks for the feedback! Keep it up

 

[ironchefpython]

I think (may be off) that the serialization note there relates to how we need to both serialize stuff for network transport (binary) as well as serialize stuff like simple settings to disk, which would be stored in plaintext

And I was suggesting that the same serialization format be used for both of those things, as well as every other piece of information associated with a world. (note that I'm not talking about settings that aren't associated with a world, such as audio or video settings) I can't imagine a situation where you wouldn't want to send world settings to the client, prior to sending chunk data and mod metadata. Or where you would load a world and not want to load both the settings and the other data.

Let's pretend we're using JSON for a moment.

{
    "settings": {
        "seed": 123456789
    },
    "player": {
        "position": ...
        "orientation": ...
        "health": ...
    },
    "chunks": [
    ...
      chunk data
    ...
    ],
    "mod1data": {
        ....
    },
    "mod2data": {
        ....
    }
}

Everything needed to model the world is there in a single file. Anything that isn't specific to a particular world, such as mods, textures, etc. would be loaded (or sent to the client) prior to loading the world.

 

[Cervator]

Hmm, I might be simplifying stuff in my head too much, or expecting current setup with single-player to survive past adopting networking. Somewhere you'll want to save something somewhere in plain text, no? I think I might be misunderstanding something, maybe that serializing to binary or plain text would require different strategies?

 

[Immortius]

I'm getting confused. Can we be clear on whether we are talking about formats or mechanisms? Because having a common mechanism for serializing things is sensible, but using a single format for everything is not.

I think network communications still need to be handled separately because different information (or more strictly, a subset) needs to be sent. To be precise, there is a whole lot of data that clients don't need to know about, but which do represent world state that needs to be persisted. AI state for instance. The inventories of other players (except what appears on their model). Not so much for chunks, although I can imagine things like stress information for calculating what blocks will fall off doesn't need to be replicated.

Incidentally if the NoSQL database stuff I'm experimenting with turns out to work well, then that could become our single mechanism for storage - the one I'm looking at supports dumping the database to disk as a json format, along with fun things like viewing and modifying through a web admin tool. But we'll have to see.

 

[Immortius]

This article briefly describes some of the other options in the Protobuf space: Protocol Buffers, Avro, Thrift & MessagePack.

 

[Immortius]

Have some serialization working for the Entity System now. Works through protobuff, can produce a binary or ugly text - and I have an additional gson formatter working on top of the protobuff messages to produce a nice json format as shown below.

Still need to finish off deserialization, and integrate it into the game proper - currently works through a "dumpEntities" console command.

The process is a little convoluted - it uses a series of type handlers to describe how to serialize various types to and from the protocol buffer messages, and then for each component class reflection is used to construct a set of objects describing how to serialize it (this is done once per component class) to a Component message. There is also some automatic support for serializing shallow data objects.

I plan to add some code so it will use getters and setters if present. Also would be nice to give components the option to inherit an interface they can use to serialize themselves, for efficiency.

{
  "entity": [
  {
      "id": 8,
      "charactermovement": {
        "maxGroundSpeed": 5.0,
        "maxWaterSpeed": 2.0,
        "maxGhostSpeed": 5.0,
        "runFactor": 1.5,
        "jumpSpeed": 10.0,
        "groundFriction": 8.0,
        "isGhosting": false,
        "isSwimming": false,
        "isGrounded": false,
        "isRunning": false,
        "velocity": [
          0.0,
          0.0,
          0.0
        ],
        "jump": false,
        "drive": [
          -0.5551398,
          0.0,
          -0.831757
        ],
        "faceMovementDirection": true,
        "distanceBetweenFootsteps": 1.0,
        "footstepDelta": 0.0
      },
      "charactersound": {
        "footstepSounds": [
          "Slime1",
          "Slime2",
          "Slime3",
          "Slime4",
          "Slime5"
        ],
        "footstepVolume": 0.7
      },
      "aabbcollision": {
        "extents": [
          0.5,
          0.5,
          0.5
        ]
      },
      "mesh": {
        "renderType": "GelatinousCube",
        "color": [
          0.2,
          1.0,
          0.2,
          1.0
        ]
      },
      "simpleai": {
        "lastChangeOfDirectionAt": 48559529,
        "movementTarget": [
          -486.31122,
          0.0,
          -639.33374
        ],
        "followingPlayer": false
      },
      "location": {
        "position": [
          -221.92918,
          0.0,
          -243.21442
        ],
        "rotation": [
          -0.0,
          -0.95701545,
          -0.0,
          0.29003698
        ],
        "scale": 0.29767874
      }
    },
    {
      "id": 7,
      "item": {
        "name": "Railgun",
        "icon": "Railgun",
        "stackId": "",
        "stackCount": 1,
        "renderWithIcon": false,
        "usage": "InDirection",
        "consumedOnUse": false,
        "baseDamage": 1
      },
      "tunnelaction": {},
      "playsoundaction": {
        "sounds": [
          "Explode1",
          "Explode2",
          "Explode3",
          "Explode4",
          "Explode5"
        ],
        "volume": 1.0
      }
    },
    {
      "id": 6,
      "item": {
        "name": "Dynamite",
        "icon": "Dynamite",
        "stackId": "",
        "stackCount": 1,
        "renderWithIcon": false,
        "usage": "OnBlock",
        "consumedOnUse": false,
        "baseDamage": 1
      },
      "explosionaction": {},
      "playsoundaction": {
        "sounds": [
          "Explode1",
          "Explode2",
          "Explode3",
          "Explode4",
          "Explode5"
        ],
        "volume": 1.0
      }
    },
    {
      "id": 5,
      "item": {
        "name": "PickAxe",
        "icon": "PickAxe",
        "stackId": "",
        "stackCount": 1,
        "renderWithIcon": true,
        "usage": "None",
        "consumedOnUse": false,
        "baseDamage": 1,
        "perBlockDamageBonus": {
          "Stone": 1
        }
      }
    },
    {
      "id": 4,
      "item": {
        "name": "Axe",
        "icon": "Axe",
        "stackId": "",
        "stackCount": 1,
        "renderWithIcon": true,
        "usage": "None",
        "consumedOnUse": false,
        "baseDamage": 1,
        "perBlockDamageBonus": {
          "BirkTrunk": 1,
          "OakTrunk": 1,
          "PineTrunk": 1
        }
      }
    },
    {
      "id": 3,
      "light": {},
      "blockitem": {
        "blockGroup": "Torch"
      },
      "item": {
        "name": "Torch",
        "icon": "",
        "stackId": "TorchBlock",
        "stackCount": 99,
        "renderWithIcon": false,
        "usage": "OnBlock",
        "consumedOnUse": true,
        "baseDamage": 1
      }
    },
    {
      "id": 2,
      "blockitem": {
        "blockGroup": "Companion"
      },
      "item": {
        "name": "Companion",
        "icon": "",
        "stackId": "CompanionBlock",
        "stackCount": 16,
        "renderWithIcon": false,
        "usage": "OnBlock",
        "consumedOnUse": true,
        "baseDamage": 1
      }
    },
    {
      "id": 1,
      "charactermovement": {
        "maxGroundSpeed": 3.0,
        "maxWaterSpeed": 2.0,
        "maxGhostSpeed": 5.0,
        "runFactor": 1.5,
        "jumpSpeed": 10.0,
        "groundFriction": 16.0,
        "isGhosting": false,
        "isSwimming": false,
        "isGrounded": true,
        "isRunning": false,
        "velocity": [
          0.0,
          0.0,
          0.0
        ],
        "jump": false,
        "drive": [
          0.0,
          0.0,
          0.0
        ],
        "faceMovementDirection": false,
        "distanceBetweenFootsteps": 1.5,
        "footstepDelta": 0.0
      },
      "charactersound": {
        "footstepSounds": [
          "FootGrass1",
          "FootGrass2",
          "FootGrass3",
          "FootGrass4",
          "FootGrass5"
        ],
        "footstepVolume": 1.0
      },
      "aabbcollision": {
        "extents": [
          0.3,
          0.8,
          0.3
        ]
      },
      "player": {
        "spawnPosition": [
          -24340.0,
          54.0,
          -24525.0
        ]
      },
      "inventory": {
        "itemSlots": [
          2,
          3,
          4,
          5,
          6,
          7,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0
        ]
      },
      "localplayer": {
        "viewYaw": 58.350006,
        "viewPitch": -15.375,
        "isDead": false,
        "respawnWait": 0.0,
        "selectedTool": 0,
        "handAnimation": 0.0
      },
      "location": {
        "position": [
          -24340.0,
          55.300003,
          -24525.0
        ],
        "rotation": [
          0.0,
          0.48747876,
          0.0,
          0.87313485
        ],
        "scale": 1.0
      },
      "health": {
        "maxHealth": 255,
        "regenRate": 5.0,
        "waitBeforeRegen": 2.0,
        "fallingDamageSpeedThreshold": 20.0,
        "excessSpeedDamageMultiplier": 10.0,
        "timeSinceLastDamage": 0.0,
        "partialRegen": 0.0,
        "currentHealth": 255
      }
    }
  ],
  "next_entity_id": 9
}

 

[Cervator]

Spiffy - although would you really serialize individual sounds rather than just a reference to an SFX prefab or something?

(I'm probably just looking for something to comment on rather than finding a spot that isn't already planned for later)

 

[Immortius]

Spiffy - although would you really serialize individual sounds rather than just a reference to an SFX prefab or something?

That is a good question, and I guess it touches on two things.

First, the sound asset. I am reminded of the Unreal engine, as it has two types of sound assets - SoundWaves, which correspond to single sounds and which cannot be directly played, and SoundCues, which can be played. SoundCues can be composed of a number of sounds (one of which is played at random) and contain other information that alters the behaviour of the sounds. If we used a similar system then it would reduce the need for components to have a list of sounds to pick from. But it does mean an extra level to manage.

The more fundamental part is how components should be handled. Right now the entity system uses the safest, but most repetative formulation, but there are some other worth considering.

1. Components wholly owned by individual entities. This is what is done at the moment. It has the advantage of keeping modifications fast and safe - when you change a component, you know you are only affecting the entity it belongs to. There is no confusion about what will happen. It does mean a lot of repetition though for things like CharacterSoundComponent, which won't change much.

2. Components wholly owned by individual entities, but prefab linked and only deltas saved. A slight change would be to have entities linked to the prefab they were created from. For serialization only fields that differ from the prefab would need to be saved. Besides reducing the amount of information that needs to be saved, it also means that the prefabs could be updated and the changes reflected on existing entities based on that prefab. Doesn't help so much at runtime though.

3. Shared components. Components can be shared between multiple entities, or be unique to a single entity. Obviously can save a lot of repetition with this. My concern with this is that it makes it harder to manage components. Given a naive approach where you just share the component objects between entities, you can never be sure whether changing a component will affect just the entity you are working on, or a whole suite of entities. I can think of a couple of ways to address this:

a) When a shared component is requested, create a copy. If it is changed, set the entity to use the copy instead. To the user components would not appear shared.

b) Have a clear separation between sharable components and nonsharable component types, so the user knows which is which.

The other option is for common shared data, turn it into an asset rather than a component - assets are essentially immutable during gameplay and served up through flyweight-style asset managers, so only one instance of each will exist. But I wouldn't want to go overboard with them.