Block Entity Behavior

[Immortius]

I think it would be useful to have a discussion on the Block Entity behavior - what features are available and how the system should behave when different things occur like blocks changing. Once that is determined we can work to ensure the system behaves that way. @Marcin, as you have been making most use of this your feedback would be good - certainly you've been uncovering all the bugs and lackings with the current setup.

Some performance goals here are to keep as few block entities in existence as possible, because it is otherwise quite easy to overwhelm memory with them. If only 1% of blocks have entities that is still tens of thousands of them.

Here's a starting point. It includes concepts that are only in multiplayer at the moment since I consider that the direction things are heading:

Entity Lifecycle

Entities have the following lifecycle

Entities are created, after which they are active. Activate entities may later be destroyed.

Irrelevant entities are subject to being stored (deactivated). This is determined by the engine, and may occur or not occur as it sees fit.

An entity reference may then have three states: Non-existent (a null entity reference, or a reference to a destroyed entity), Stored (exists but not in memory) and Active (exists and is available). Entities that are not active cannot be modified or destroyed, but a reference can be held to them.

Generally entity deactivation and activation follows these rules (block entities are slightly different):

• An entity can be marked as transient. A transient entity will never be persisted, and is instead destroyed when it loses relevance. Other entities are stored.
• An entity that is owned gains and loses relevance with its owner.
• An entity that is spatially attached to another gains and loses relevance with its spatial parent.
• An player's character loses relevance when its player disconnects.
• Any entity with a Location component that is unparented loses relevance when the chunk it resides is unloaded by default. This can be overridden.
• Any other entity is always relevant.
• Autosaving will persist entities without unloading them.

Component Lifecycle

Components follow the following lifecycle.

(Rounded boxes are states, rectangular boxes are events that are by the transitions)

• When an entity is created, all of its initial components are added and activated - OnAddedComponent and OnActivatedComponent events will be sent to the entity in that order.
• When the entity is destroyed its components are deactivated and destroyed - BeforeDeactivatedComponent and BeforeRemoveComponent events will be sent to the entity in that order.
• When the entity is Stored, its components are deactivated and removed from memory - a BeforeDeactivatedComponent event will be sent to the entity.
• When the entity is Loaded again, its components are activated. An OnActivatedComponent event is sent to the entity.
• An active entity can have components added - an OnAddedComponent and OnActivatedComponent event will be sent to the entity tied to the added component and in that order.
• An active entity can also have components removed - BeforeDeactivatedComponent and BeforeRemoveComponent events will be sent to the entity tied to the removed component and in that order.
• An active entity can have components changed and saved - a ChangedComponent event will be sent to the entity when this occurs.

BeforeDeactivatedComponent and BeforeRemvoedComponent are sent before the component is deactivated or removed, so the component can be retrieved and inspected from the entity by event receivers of these events.

All the component lifecycle events behave a little differently to normal events - in addition to only receiving the event if the entity involved has all of the components in the @ReceiveEvent's component list, the event will only be received if one of the components in the list is involved in the event.

Example:

@ReceiveEvent(components = {LocationComponent.class, MeshComponent.class})
public void onReadyToRender(OnAddedComponentEvent event, EntityRef entity);

This event receiver is guaranteed to only be called when an entity has both a location and mesh component, and one or both of those components was just added. It will not be called if additional components are added. It will be called again if one of those components is removed and later re-added.


Chunk Lifecycle

Chunks have the following states:

• Ungenerated - the chunk hasn't been fully generated yet. No block entities exist for it.
• Loaded- the chunk is generated and available for use.
• Stored- the chunk has been generated, but is not available for use at the moment.

For clients in multiplayer the states are only Loaded and Stored, as the client has no concept of generation nor any way to track it.

The following events are sent to the world entity (the entity with the WorldComponent) relating to the chunk lifecycle:

• OnChunkGenerated - When a chunk has just been generated.
• OnChunkLoaded - When a chunk has been loaded and is ready for use - whether it is coming from an unloaded state or from being generated.
• OnChunkUnloaded - When a chunk is about to be stored (happens before unload so blocks and block entities are available for inspection).

Block Types and Block Families

In Terasology, a block type is a unique appearance for a world location. Each distinct shape, texturing and rotation is a different block type.

A block family is a group of one or more block types with similar appearance and behavior. The simplest block families contain just a single block type, such as a dirt or stone block (rotation doesn't matter). However more complex block families can contain a number of block type variants to account for different rotations or forms. For example, a fence may be placed in a number of rotations, and may have different shapes to connect to walls or other fences.

Block families allow a group of different block types to be treated as the same item when picked up, and when that item is placed an appropriate block type is chosen for that location based on the conditions of the placement (the direction the player is looking, the surface they are targeting and the state of the surrounding blocks).

Block Type Entities

In addition to the base properties of a block type, each block type can be associated with a prefab. This prefab provides additional components that a block of that type will have on its entities (more on this below).

Each block type has an associated entity. This entity has a BlockTypeComponent, a NetworkComponent (to propagate it to clients) and each component of its prefab with the settings from the prefab, excluding the prefab's NetworkComponent if any.

(Note: In the future components may be declared inside a block, rather than requiring an external prefab.)

Block Entities

All block entities have a BlockComponent (to mark them as a block), and a LocationComponent (giving their location).

By default, blocks (each individual block in the world, as opposed to block types) do not have normal entities. Instead when they need to be interacted with a temporary entity is brought into existence to interact with, based on the blocks prefab. At the end of the frame, the temporary entity vanishes once more. When these temporary entities enter or leave existence, no component events are sent (no OnAdded, OnActivated, OnDeactivated or OnRemoved events).

Some blocks need to have entities that remain active - this allows them to be changed and be available for system update() calls. An example would be a block that is rendered by its entity, rather than through the block shape system. These block entities will also be stored and restored with the chunk they belong to.

This can be set for each block in the block's definition. Additionally, components can be marked with @ForceBlockActive - if a component so marked is added to a block entity then the block's entity will remain active. When all @ForceBlockActive components are removed the block will switch being temporary. For instance, when a block is damaged it is given a DamagedBlockComponent with the @ForceBlockActive annotation - as long as the block is damaged its entity will remain active (and the block damage effect rendered).

Behavior when a chunk is generated/unloaded/loaded

When a new chunk is generated and becomes Active, the following things occur:

1. For each block that is needs to be active continuously, the entity for the block is created. Standard OnAddedComponent and OnActivatedComponent events are sent to these entities.
2. Every other block its location added to a list of blocks for that type. Then an OnBlockAdded event is sent to each Block Type entity with its list of block locations - this allows a system to know all of the blocks that have been added of that type. Then an OnBlockActivated event is sent to each Block Type entity with the same lists. This allows for batch processing of blocks with a desired component in their prefab.

When a chunk is about to be unloaded, the following things occur:

1. For each active block, the entity for the block is stored. Standard BeforeDeactivatedComponent events are sent to these entities.
2. Every other block its location added to a list of blocks for that type. Then an OnBlockDeactivated event is sent to each Block Type entity with its list of locations.

When a chunk is loaded the process is similar:

1. Entities within the chunk are restored and reactivated. This includes all block entities that were active when stored. Standard OnActivatedComponent events are sent to these entities.
2. Each block without an active entity has its location added to a list of blocks for that type. Then an OnBlockActivated event is sent to each Block Type entity with its list of locations.

Behavior When Blocks Changed

In general, any change to a block will attempt to modify the block's entity rather than remove it and create a new entity. Examples where this occurs:

• A dirt block is replaced with grass. The temporary block entity for the dirt block will be modified to become grass, changing to match grass' prefab. Standard component lifecycle events will be sent for these changes.
• An active block is replaced with a temporary block. The existing entity will be updated and become a temporary block entity.

Retained Components

Components marked with the @RetainWhenBlockRemoved annotation are copied onto Dropped Block and Block Item components, and copied back onto block entities when the block is placed. This annotation also has the same effect as a @ForceBlockActive annotation.

(Block Groups

Block groups are entities sit over a collection of blocks - an example is a door, which has an entity that sits over both the upper and lower door. When an entity is requested for a block, a block group if any is provided first. Interactions such as damage are sent to the block group first

(TODO: fill this section in more))

Future thoughts:

• Add BeforeStore and OnRestore events to be sent against entities being stored/restored - but only when a use case emerges for this.

 

[Immortius]

I can see why you want to avoid having persistent entities for blocks unless really needed (like chests). But then you are presented with 2 choices, you either have behavior steered by block type or by its components.
The first choice requires for systems to work off the BlockChanged events, however if a behavior is to be added to a block, it has to be replaced with a different one, which seems to be the way Terasology is intended to work currently. Lets say, I ionize air for a period of time and while it's ionized, it should conduct electricity. To achieve that, I'd have to replace the air block with "ionized air" block added in my mod. This would however change how other systems work, as they might be checking for "air" (like water spreading) and not find it there anymore. Simply adding a "conducts electricity" component to the air block does not bode well with the system, as that means that my system would have to add AddComponent and RemovedComponent event listeners as well, side effect of that is, that these would be called randomly due to entities being created for the duration of a tick on a user interaction as it is currently.

If another system is checking for "air" then I would say it is wrong - it should be checking either a property of the block type or its entity/components. For instance, it should check for non-solid blocks. We need to do some work in this space - either changing the base properties of blocks, adding support for custom block properties or adding more information to our existing blocks. We probably also need a way of adding a property across existing blocks - so a mod can introduce a new property and set it for blocks defined in the engine.

That aside, I agree AddComponent and RemovedComponent events are of limited use for blocks - this doesn't really concern me though because they are low level (mostly for internal engine use) and it is intended for higher level events to be introduced as desired. You could send an IonizedEvent for instance. I think the OnBlockTypeChanged, OnBlockEntityReplaced and OnBlockFamilyChanged events above should help though? Maybe OnBlockEntityChanged need to be sent when a non-temporary block has an entity changed too, though?

 

[Marcin Sciesinski]

I'm at the airport now, so just a quick comment, will write more later on.

It would be nice to flesh out, what lifecycle events are generated for on_interaction and while_placed block entities. So, if I have on_interaction block entity, is it going to trigger OnAddedEvent for each component each time it's going to be created, or only when it is placed for the first time?

So, for example:
"on_interaction" block entities generate following events:
OnAddedEvent - when placed for the first time in the world,
...
"while_placed" block entities generate following events:
...

Also, I have a feeling that the OnStoreEvent should be replaced by OnBeforeStoreEvent and OnAfterStoreEvent. I have a vague memory of some other framework using it this way - the intent was, that the OnBeforeStoreEvent could clean up any transient data from the entity, that doesn't need to be stored. Unless you are planning to add transient support to the engine itself, though it might get way too complicated, as some data might be transient in some cases, and in other cases it has to be persisted, so it's better to leave it up to the System responsible to take care of this.

 

[Immortius]

I'm at the airport now, so just a quick comment, will write more later on.

It would be nice to flesh out, what lifecycle events are generated for on_interaction and while_placed block entities. So, if I have on_interaction block entity, is it going to trigger OnAddedEvent for each component each time it's going to be created, or only when it is placed for the first time?

So, for example:
"on_interaction" block entities generate following events:
OnAddedEvent - when placed for the first time in the world,
...
"while_placed" block entities generate following events:
...

Sure. Basically OnAddedEvent + OnActivatedEvent would be sent whenever an ON_INTERACTION block entity is created, and OnDeactivatedEvent + OnDestroyedEvent would be sent whenever it is cleaned up again. For WHILE_PLACED, they bracket the block being placed. This falls out of the ON_INTERACTION block entities being temporary, and those events being the domain of the entity system. I'll add this in.

Also, I have a feeling that the OnStoreEvent should be replaced by OnBeforeStoreEvent and OnAfterStoreEvent. I have a vague memory of some other framework using it this way - the intent was, that the OnBeforeStoreEvent could clean up any transient data from the entity, that doesn't need to be stored. Unless you are planning to add transient support to the engine itself, though it might get way too complicated, as some data might be transient in some cases, and in other cases it has to be persisted, so it's better to leave it up to the System responsible to take care of this.

The framework currently supports both transient entities (by setting the entity to not be persisted via the entity ref), and transient fields (via the standard java transient keyword), although I'm not certain transient fields are a good idea - in particular if the entity system later defensive copies components (which would enable some nice features) or is backed by an in memory database then transient fields are essentially useless. Or alternatively transient data can be stored in systems, as they are never persisted. In general every component and every field in each component is persisted though.

I'm actually thinking of removing OnStoreEvent now. It was needed before the @Owns annotation was set up, to allow systems to add extra entities into the store during the process of storage (so adding the items contained in an inventory). Since @Owns allows a component to specify this relationship it isn't needed now. I can see the need for a OnBeforeStoreEvent, but the entity cannot receive events any more after being stored so an OnAfterStoreEvent may be tricky.

 

[Marcin Sciesinski]

Another airport and another comment...

I see that for you "store" is when the entity (and chunk) is stored in store and removed from memory, while I understood it's when an entity (and chunk) is stored, but not necessarily being unloaded, for example an auto-save feature.

I'm also wondering about the change of persistence mode for entities. In my example with ionizing air, while the air either doesn't have an entity at all, or has an "on_interaction" entity, the component that is added to specify it's ionized should change this block entity to be "while_placed". Once the component is removed, it should revert again to the "on_interaction" mode. This suggests, each entity should keep a counter, and increment each time a component is added that requires to change its mode, and decrement when the component is removed. Of course this should be done explicitly by the system adding/removing the component. The resulting mode should depend on number of those "tickets" registered. If an entity has at least one of them, it has to be persisted.

 

[Immortius]

Ah, I see. I'll clarify that - I specifically meant the unloading more than the saving. In theory the entity system could be backed by an in-memory database that automatically saves to disk,which would not require an explicit autosave. In practice we probably will end up with an autosave though - but I don't think systems need to get involved in that.

On changing the persistence mode of block entities - the way I would probably implement it is having an annotation like @PersistsBlock that is applied to a component, with the block entity system keeping track behind the scene - that way each system won't have to mess with counters or whatnot (which leaves scope for modder error). Shouldn't actually need tickets, can just iterate over the block entity's components and see if any have that annotation.

 

[Marcin Sciesinski]

For me the onAdded and onActivated being sent, when an "on_interaction" entity is created due to interaction (say, someone looking at it) is still a "no go". The reason is, that the same events would be triggered, when a block is placed, yet these two actions are completely different from every point of view.

I will be suggesting an alternative approach. On the plane I was writing a TDD (technical design document) including some diagrams. Let me finish it off and clean it up and I will post it in this thread.

 

[Marcin Sciesinski]

I'd like to suggest a little bit different approach. I wrote this TDD on the plane. Let me know what you think.

Block state diagram

Blocks have 3 states:

1. Does not exist - the block at specific position has not been generated yet.
2. Active - block at that position has been generated already and is currently active (loaded in memory).
3. Unloaded (inactive) - block at that position has been generated already, but it's not currently in memory.

Between these 3 states there are 4 possible transitions:

1. When a block is generated, it transitions between "Does not exist" and "Active" state. During this transition an "onGenerate" event is fired.
2. When a chunk is unloaded (no longer needed, game exits, etc), it transitions between "Active" and "Unloaded" state. During this transition an "onUnload" event is fired.
3. When a chunk is loaded (game starts, player moves to a location close to the block, etc), it transitions between "Unloaded" and "Active" state. During this transition an "onLoad" event is fired.
4. When a system replaces a block at the position, it does not change its state, however an "onChange" event is fired (feel free to add different flavors, like FamilyChange, BlockChange, etc).

Component state diagram

Components of an entity have 4 states:

1. Not defined - this entity doesn't have this component defined.
2. Active - this entity has this component defined and it is active.
3. Unloaded - this entity has this component defined, but it's not currently active (unloaded).
4. Garbage collected - this entity has this component defined it's active, but it's not currently stored in memory. Note: this has nothing to do with JVM garbage collection.

Between these 4 states there are 7 possible transitions:

1. When a block is changed and the new block has a component in its "prefab" configuration, or when a component is saved to an entity that didn't have this component before, component transitions between "Not defined" and "Active" state. At this time the "onAdd" event is fired.
2. When a chunk is unloaded, component transitions between "Active" and "Unloaded" state. At this time the "onDeactivate" event is fired.
3. When a chunk is loaded, component transitions between "Unloaded" and "Active" state. At this time the "onActivate" event is fired.
4. At the end of a tick, if the entity is in "transient" (doesn't need to be stored), it transitions between "Active" and "Garbage collected" state and is removed from memory. At this time the "onDestroy" event is fired. (this event is optional, might be used for debugging purposes only)
5. If any system queries for the entity and it has this component in its prefab, it transitions between "Garbage collected" and "Active" state. At this time the "onRestore" event is fired. (this event is optional, might be used for debugging purposes only)
6. If a component is saved to this entity, and the entity already had this component defined, it does not change its state, however the "onChange" event is fired.
7. When a component is removed from the entity, or the block is removed (replace with different block) it transitions between "Active" and "Not defined" state. At this time the "onRemove" event is fired.

Entity persistence mode
There are 3 entity persistence modes, which define how, when and if an entity is stored in memory and/or on disk.

1. Transient (default where applicable) - entity is not stored on disk and is guaranteed (the minimum) to be stored only until the end of the current tick. Systems should not depend on the fact, that the entity is going to disappear at the end of the tick, as the core software can be optimized to store those for a longer duration if multiple queries for the same entities are detected.
2. Chunk_loaded - entity is stored in storage with the chunk it is defined in. This persistence mode only applies to entities that have LocationComponent. If an entity is detected with "Chunk_loaded" persistence mode, but no LocationComponent, it should be considered as having the "Always" persistence mode. When the chunk is unloaded, the entity is also removed from memory.
3. Always - entity is stored in general storage for the whole world. This entity is always in memory when the world is loaded.

Two approaches of determining entity persistence modes are suggested, merits of both should be discussed:

1. Each component, in its definition defines its persistence mode using a Java annotation. Entity's persistence mode is the minimum required for all its components.
2. Rather than each component defining its persistence mode in its definition. Whenever a component is added to an entity (either via code, or using prefab), an extra parameter is required of the 3 possible values (transient, chunk_loaded, always). Entity's persistence mode is the minimum required for all its current components.

Second approach requires a bit more work, but adds flexibility. If an entity's component has default values (as defined in prefab), it might require "transient" persistence mode, as it could be recreated from prefab at any time. But if any of the values is changed, it might be upgraded to "chunk_loaded" (or "always") during a save component call, so only entities that have modified state will have to be stored. When a system reverts the value to the default in the component for the entity it might downgrade the persistance mode back to "transient".
Similar optimization can be made for first approach, however the framework itself would have to detect this situation (entity having default values for all components). This could also prevent systems depending on iterating over all entities with a specified component (assuming they are active), while the framework has "optimized" those entities and Garbage Collected them. This could lead to hard to find and debug problems.
Additional notes

1. Feel free to change names of the states as well as events.
2. All events should be called AFTER the operation they describe is made (added, removed, changed, etc). If there is a need for event fired before some transition an "onBefore..." naming convention should be used. Only the "onBefore..." events should be cancellable, as you can't cancel something that has already happened.
3. Only if the block is in an Active state and a component is in an Active state can it be queried via iterating methods of the EntityManager (or whatever the class name is called).

 

[Immortius]

For me the onAdded and onActivated being sent, when an "on_interaction" entity is created due to interaction (say, someone looking at it) is still a "no go". The reason is, that the same events would be triggered, when a block is placed, yet these two actions are completely different from every point of view.

They're the same from the point of view of the entity system - since in both cases the entity is being created. However one way of dealing with this is to pseudo-store/restore ON_INTERACTION blocks - then you would only get OnAdded/OnRemoved at the beginning and end of the block's life cycle - would still get OnActivated/OnDeactivated throughout the lifecycle of the block, but that should be fine?

TDD

1. Does not exist - the block at specific position has not been generated yet.
2. Active - block at that position has been generated already and is currently active (loaded in memory).
3. Unloaded (inactive) - block at that position has been generated already, but it's not currently in memory.

These are all chunk states, essentially (and block states by implication). They are the states we have at the moment, but only the chunk provider knows about the difference between the 1st and 3rd states - for every other system there's no difference between an unloaded and ungenerated chunk. Do you see an issue with this, or does this need to be exposed?

1. When a block is generated, it transitions between "Does not exist" and "Active" state. During this transition an "onGenerate" event is fired.
2. When a chunk is unloaded (no longer needed, game exits, etc), it transitions between "Active" and "Unloaded" state. During this transition an "onUnload" event is fired.
3. When a chunk is loaded (game starts, player moves to a location close to the block, etc), it transitions between "Unloaded" and "Active" state. During this transition an "onLoad" event is fired.
4. When a system replaces a block at the position, it does not change its state, however an "onChange" event is fired (feel free to add different flavors, like FamilyChange, BlockChange, etc).

1. would seems to imply every single block in a chunk (64k of them) would be sent an onGenerate event. But I assume what you really mean is the chunk receives the event (or the world, since we don't have chunk entities at the moment). I'ld rename it onGenerated.

2. This is fine. I'ld change it to onUnload being called just prior to unloading (since that is when it becomes inaccessible).

3. Also fine, I'ld change it to onLoaded being called after loading.

4. This is doable - it means sacrificing having before and after entities, but that is fine. So essentially when a block is replaced, at worst the entity of the old block is modified with components added, changed and removed to become the new entity based on the prefab.

1. Not defined - this entity doesn't have this component defined.
2. Active - this entity has this component defined and it is active.
3. Unloaded - this entity has this component defined, but it's not currently active (unloaded).
4. Garbage collected - this entity has this component defined it's active, but it's not currently stored in memory. Note: this has nothing to do with JVM garbage collection.

...

1. When a block is changed and the new block has a component in its "prefab" configuration, or when a component is saved to an entity that didn't have this component before, component transitions between "Not defined" and "Active" state. At this time the "onAdd" event is fired.
2. When a chunk is unloaded, component transitions between "Active" and "Unloaded" state. At this time the "onDeactivate" event is fired.
3. When a chunk is loaded, component transitions between "Unloaded" and "Active" state. At this time the "onActivate" event is fired.
4. At the end of a tick, if the entity is in "transient" (doesn't need to be stored), it transitions between "Active" and "Garbage collected" state and is removed from memory. At this time the "onDestroy" event is fired. (this event is optional, might be used for debugging purposes only)
5. If any system queries for the entity and it has this component in its prefab, it transitions between "Garbage collected" and "Active" state. At this time the "onRestore" event is fired. (this event is optional, might be used for debugging purposes only)
6. If a component is saved to this entity, and the entity already had this component defined, it does not change its state, however the "onChange" event is fired.
7. When a component is removed from the entity, or the block is removed (replace with different block) it transitions between "Active" and "Not defined" state. At this time the "onRemove" event is fired.

See my above suggest on pseudo-storing transient block entities - I think that Garbage Collected state is unnecessary. Transient block entities would be unloaded into nothing at the end of the tick, and reloaded from prefab when needed. You would still have the OnLoad/OnUnload events for the chunk as a whole, and the guarantee that OnAdded/OnRemoved is only called when the block's entity has new components added and removed.

There are 3 entity persistence modes, which define how, when and if an entity is stored in memory and/or on disk.

1. Transient (default where applicable) - entity is not stored on disk and is guaranteed (the minimum) to be stored only until the end of the current tick. Systems should not depend on the fact, that the entity is going to disappear at the end of the tick, as the core software can be optimized to store those for a longer duration if multiple queries for the same entities are detected.
2. Chunk_loaded - entity is stored in storage with the chunk it is defined in. This persistence mode only applies to entities that have LocationComponent. If an entity is detected with "Chunk_loaded" persistence mode, but no LocationComponent, it should be considered as having the "Always" persistence mode. When the chunk is unloaded, the entity is also removed from memory.
3. Always - entity is stored in general storage for the whole world. This entity is always in memory when the world is loaded.

I don't think the transient block feature needs to be brought into the general entity system (doesn't make sense for any other entities, as what would "restore" them as needed?). The (1) mode would be unpersisted, which means it will not be saved to disk.

Second approach requires a bit more work, but adds flexibility. If an entity's component has default values (as defined in prefab), it might require "transient" persistence mode, as it could be recreated from prefab at any time. But if any of the values is changed, it might be upgraded to "chunk_loaded" (or "always") during a save component call, so only entities that have modified state will have to be stored. When a system reverts the value to the default in the component for the entity it might downgrade the persistance mode back to "transient".
Similar optimization can be made for first approach, however the framework itself would have to detect this situation (entity having default values for all components). This could also prevent systems depending on iterating over all entities with a specified component (assuming they are active), while the framework has "optimized" those entities and Garbage Collected them. This could lead to hard to find and debug problems.

The annotation actually allows for the changing of default values situation - the system could throw a marker component with the annotation on the block when it makes a change, and remove it when it wants to revert the block to default.

1. Feel free to change names of the states as well as events.
2. All events should be called AFTER the operation they describe is made (added, removed, changed, etc). If there is a need for event fired before some transition an "onBefore..." naming convention should be used. Only the "onBefore..." events should be cancellable, as you can't cancel something that has already happened.
3. Only if the block is in an Active state and a component is in an Active state can it be queried via iterating methods of the EntityManager (or whatever the class name is called).

On (2), removed events are always fired before the component is removed, otherwise you wouldn't get a chance to inspect the removed component (and you couldn't subscribe to the event anyway). On the cancel thing, that just prevents other event receivers getting the event, it doesn't stop the action occurring - for these lifecycle events the cancel method probably should throw an UnsupportedMethodException or similar.


The one big issue remaining I see is this: should OnAddedComponent methods be sent for transient blocks when a chunk is first generated? Or perhaps just transient blocks with a prefab? Or perhaps there needs to be an extra flag for this?

 

[Immortius]

1. When a system replaces a block at the position, it does not change its state, however an "onChange" event is fired (feel free to add different flavors, like FamilyChange, BlockChange, etc).

Oh, another thing I missed - how does this work with "detatched" blocks? For instance, when a player picks up a chest block, at the moment the chest block's entity ceases to be a block entity and is attached to the held chest item entity - so that it still has the same inventory when later placed again. Obviously in this case the old entity cannot be changed, instead a new block entity is required.

 

[Marcin Sciesinski]

What I was getting to, was that with the current state of states and events it's impossible to do anything more complicated. Systems are unable to track which blocks with a specific component are currently "active" (either loaded or "garbage collected" but in active chunks that have the component in prefab).

On the matter of your comments - I don't think "content" systems would ever use block events, as behaviors should be driven by components, only some "core" systems will probably use block events (for example BlockFamily replacing blocks to form correct block connections). As a matter of fact, we can make the block events a "core" functionality and discourage using them in "content" systems. If it comes to that, we can introduce the events as we go and they become required.

Anyway on to the questions:

They are the states we have at the moment, but only the chunk provider knows about the difference between the 1st and 3rd states - for every other system there's no difference between an unloaded and ungenerated chunk. Do you see an issue with this, or does this need to be exposed?

None of the states have to be exposed in any way. States are just a logical terms to figure out, how an object (block/component) transitions throughout its life-cycle. As an example, there is no need for a method like "public BlockState getBlockState(Vector3i location)". However, I think there is a value in having different events for transitions between different states (if we decide to add them, as stated above). To give an example off the top of my head - it would be possible to write a system that tracks an amount of a specific block in the whole world (loaded and unloaded), as long as the system would persist its counter between application restarts. Is there any value in this - probably not, therefore I'd introduce the events only if we find any need for them.

This is doable - it means sacrificing having before and after entities, but that is fine. So essentially when a block is replaced, at worst the entity of the old block is modified with components added, changed and removed to become the new entity based on the prefab.

Once we get the component events sorted out, there would be no need for before and after entities. In fact, as we have already seen, sometimes it's beneficial to replace a block retaining the entity. So, keep block events separately from component events, to the point that block events might become private contract rather than public.

You would still have the OnLoad/OnUnload events for the chunk as a whole, and the guarantee that OnAdded/OnRemoved is only called when the block's entity has new components added and removed.

Just want to make 100% sure. OnAdded, OnRemoved, OnLoad, OnUnload will not be called when transient entity is created from prefab during an interaction, or destroyed at the end of a tick.

I don't think the transient block feature needs to be brought into the general entity system (doesn't make sense for any other entities, as what would "restore" them as needed?).

True, transient applies to blocks only, as blocks are persisted in addition to their entities, and they are the ones that know, that there "is" a transient entity, this is not true for any other objects.

The annotation actually allows for the changing of default values situation - the system could throw a marker component with the annotation on the block when it makes a change, and remove it when it wants to revert the block to default.

This is not going to fix the problem. What if two separate systems add one component each, and due to these components they want to have a higher persistence level. If they both add a marker component, and then one of them removes it (as it no longer needs to keep the entity around), then the entity will be "garbage collected" even though the other system will still assume it should be around.

On (2), removed events are always fired before the component is removed, otherwise you wouldn't get a chance to inspect the removed component (and you couldn't subscribe to the event anyway). On the cancel thing, that just prevents other event receivers getting the event, it doesn't stop the action occurring - for these lifecycle events the cancel method probably should throw an UnsupportedMethodException or similar.

Then I think it's beneficial to rename the event to onBeforeRemoved, though it's just a matter of taste. I'm not sure, what is the use case for cancel then? If it's just to stop processing further event handlers, maybe it should be called "consume"? Also, while we are on a subject of event handling priorities - I really dislike this model, as it is quite limiting, but this is probably once again a matter of taste.
The model I prefer is, the code that does a Foo action, fires "onBeforeFoo" event, that can be cancelled (vetoed) or have it's parameters modified by registered listeners, after all listeners had a chance to respond, the code checks if the event was cancelled, and if it was not, does the Foo action (taking into account modified parameters from the event) and afterwards fires "onFoo" events. To follow with an example:
"DamageSystem" would fire "onBeforeDamage" event with a parameter "damage=10". "ArmorSystem" will receive the event and modify the damage parameter to "damage=4". Upon finishing processing listeners, "DamageSystem" substracts 4 from health and fires "onDamage" with parameter "damage=4", "originalDamage=10". "ArmorSystem" picks up this event, and substracts 6 (10-4) from the armor durability.
Another example:
"DamageSystem" would fire "onBeforeDamage" event with a parameter "damage=10". "ArmorSystem" will receive the event and modify the damage parameter to "damage=4". "InvulnerabilitySystem" will receive this event as well, and due to player having an "invulnerable" effect on him, cancels the event altogether. Upon finishing processing listeners, "DamageSystem" notices the event was cancelled and does not apply the health change and does not fire the "onDamage".
And another one:
"BlockSystem" would fire "onBeforeBlockHarvested" event with a parameter "player=MarcinSc" and the block location, type, etc. "ControlZoneSystem" picks up the event and notices that the block that is to be harvested is not in the area controlled by the player "MarcinSc" and cancels the event. When the listeners are finished processing, "BlockSystem" notices the event was cancelled and does not destroy it (replace with air). As a result "HarvestSystem" does not get "onBlockHarvested" event and does not drop a block as an item, the "ToolDurabilitySystem" does not reduce the durability of a pickaxe that was used, etc.

 

[Immortius]

What I was getting to, was that with the current state of states and events it's impossible to do anything more complicated. Systems are unable to track which blocks with a specific component are currently "active" (either loaded or "garbage collected" but in active chunks that have the component in prefab).

Agreed.

None of the states have to be exposed in any way.

This is what I expected, but thought I'ld check.

Once we get the component events sorted out, there would be no need for before and after entities. In fact, as we have already seen, sometimes it's beneficial to replace a block retaining the entity. So, keep block events separately from component events, to the point that block events might become private contract rather than public.

True.

Just want to make 100% sure. OnAdded, OnRemoved, OnLoad, OnUnload will not be called when transient entity is created from prefab during an interaction, or destroyed at the end of a tick.

Correct. OnLoad and OnUnload are chunk events so wouldn't be sent to the blocks anyway. OnAdded and OnRemoved would not be sent either as the entity system would be tricked out of it.

This is not going to fix the problem. What if two separate systems add one component each, and due to these components they want to have a higher persistence level. If they both add a marker component, and then one of them removes it (as it no longer needs to keep the entity around), then the entity will be "garbage collected" even though the other system will still assume it should be around.

Sorry, I don't think I explained well enough - each system can have its own independent marker components. So a damaged block will have an DamagedComponent, and an ionized block would have an IonizedComponent. A damaged, ionized block would have both. Only when all components with the annotation are removed would the entity be "garbage collected". Alternatively the BlockComponent could have a set of strings/tokens that work the same way, but that would have less visibility (couldn't search for all blocks with those markers).

Then I think it's beneficial to rename the event to onBeforeRemoved, though it's just a matter of taste. I'm not sure, what is the use case for cancel then? If it's just to stop processing further event handlers, maybe it should be called "consume"? Also, while we are on a subject of event handling priorities - I really dislike this model, as it is quite limiting, but this is probably once again a matter of taste.

Fair enough on the naming suggestions. Cancel was named before I had a good grip on how events would be used - it sort of makes sense for something like a Damage event, not so much for other things. For input events cancel is actually overloaded as consume anyway. It probably doesn't belong on the core interface regardless - I might move it to a ConsumableEvent interface, so events can be made consumable or not as makes sense.

I'm not particularly happy with the current priority system either - another early attempt. Would it be better to have event receivers declare a list of systems they should be prioritised before/after? Or do you have a suggestion for an alternate model for determining order? (e.g. @ReceiveEvent(components = {blah}, before={PlayerDamageSystem.class},after={ArmourSystem.class}))

The model I prefer is, the code that does a Foo action, fires "onBeforeFoo" event, that can be cancelled (vetoed) or have it's parameters modified by registered listeners, after all listeners had a chance to respond, the code checks if the event was cancelled, and if it was not, does the Foo action (taking into account modified parameters from the event) and afterwards fires "onFoo" events. To follow with an example:
"DamageSystem" would fire "onBeforeDamage" event with a parameter "damage=10". "ArmorSystem" will receive the event and modify the damage parameter to "damage=4". Upon finishing processing listeners, "DamageSystem" substracts 4 from health and fires "onDamage" with parameter "damage=4", "originalDamage=10". "ArmorSystem" picks up this event, and substracts 6 (10-4) from the armor durability.
Another example:
"DamageSystem" would fire "onBeforeDamage" event with a parameter "damage=10". "ArmorSystem" will receive the event and modify the damage parameter to "damage=4". "InvulnerabilitySystem" will receive this event as well, and due to player having an "invulnerable" effect on him, cancels the event altogether. Upon finishing processing listeners, "DamageSystem" notices the event was cancelled and does not apply the health change and does not fire the "onDamage".
And another one:
"BlockSystem" would fire "onBeforeBlockHarvested" event with a parameter "player=MarcinSc" and the block location, type, etc. "ControlZoneSystem" picks up the event and notices that the block that is to be harvested is not in the area controlled by the player "MarcinSc" and cancels the event. When the listeners are finished processing, "BlockSystem" notices the event was cancelled and does not destroy it (replace with air). As a result "HarvestSystem" does not get "onBlockHarvested" event and does not drop a block as an item, the "ToolDurabilitySystem" does not reduce the durability of a pickaxe that was used, etc.

Those examples are sensible. I probably wouldn't make entity lifecycle events cancellable though.

 

[Marcin Sciesinski]

Just want to make 100% sure. OnAdded, OnRemoved, OnLoad, OnUnload will not be called when transient entity is created from prefab during an interaction, or destroyed at the end of a tick.

Correct. OnLoad and OnUnload are chunk events so wouldn't be sent to the blocks anyway. OnAdded and OnRemoved would not be sent either as the entity system would be tricked out of it.

I meant OnActivate/OnDeactivate from my diagram (or whatever they are going to be called), not OnLoad and OnUnload.

Sorry, I don't think I explained well enough - each system can have its own independent marker components. So a damaged block will have an DamagedComponent, and an ionized block would have an IonizedComponent. A damaged, ionized block would have both. Only when all components with the annotation are removed would the entity be "garbage collected". Alternatively the BlockComponent could have a set of strings/tokens that work the same way, but that would have less visibility (couldn't search for all blocks with those markers).

To make sure I understand what you mean:
Lets say I have a switch (lever) block that has SignalEmitterComponent in prefab with the default "emit: false". The entity for this block is transient, as we can easily recreate it when needed from the prefab. This component does not have the "marker annotation". However, if someone switches the lever, the property has to be set to "emit: true". The system that does that, also has to add additional component - say LeverSwitchedCompoent that does have the "marker annoration". When the lever will be switched again, the system will have to remove the component. Is that what you mean? What is the memory footprint of storing an extra component (both in memory and on disk when chunk unloaded), is it much?

Or do you have a suggestion for an alternate model for determining order? (e.g. @ReceiveEvent(components = {blah}, before={PlayerDamageSystem.class},after={ArmourSystem.class}))

Ideally, systems shouldn't have to know about each other. While this is probably doable with some careful events design, you can't predict every interaction in advance, and in those cases (until next core release version) this probably would have to be possible. To give an example:
Version 1.0 of Terasology emits "onBeforeDamage" and "onDamage" events, as described in my earlier example.
Someone sits down and writes an ArmorSystem that introduces an effect that reduces each damage taken by player wearing an armor by 2 damage. So, during the "onBeforeDamage" reduces the damage property by 2.
Someone else decides, "that's cool, I'll add a PotionSystem" that will reduce damage taken by 50%. So, during "onBeforeDamage" it just halves the damage property. Now, depending on the order the listeners will be called, when player is supposed to take 4 damage, he would either take 1 damage (ArmorSystem called first, PotionSystem called second), or 0 damage (PotionSystem called first, ArmorSystem called second). To remove this randomness, the "newer" system would have to define how to resolve this conflict, so in the annotation it will say "before/after={ArmorSystem.class}", depending how it should be resolved.
In version 1.1 of Terasology we acknowledge problems moders had with the prevention mechanism, and create a DamagePreventionSystem that is listening to "onBeforeDamage" and it emits 2 events in succession "onFixedPrevention" and "onPercentagePrevention", which allow other systems to specify how many fixed damage they prevent, and what percentage of damage they prevent. After all listeners for these 2 events are executed, it will apply the correct arithmetic and reduce the damage in "onBeforeDamage", and the prevention will be resolved correctly.
Nice thing about this incremental complexity approach is, that it is sort of backward-compatible, for example - until ArmorSystem is upgraded to version 1.1, it will still react to the "onBeforeDamage" and correctly apply the prevention, the only problem is, it might do it out-of-order, but it might have been doing that in 1.0 anyway. Once ArmorSystem upgrades to version 1.1, it will apply the prevention in correct order.

 

[Immortius]

I meant OnActivate/OnDeactivate from my diagram (or whatever they are going to be called), not OnLoad and OnUnload.

OnActivate/OnDeactivate would be called when they blind in and out of existence. I consider this desirable, because otherwise systems that depend on that behavior won't be able to work. I'm only willing to subvert the entity system so much, and fundamentally you have entities coming in and out of existence.

Does this cause an issue? I would have assumed your logic would work on OnAdded/BeforeRemoved events, and store at least some network information in entities that are persisted with the chunks.

To make sure I understand what you mean:
Lets say I have a switch (lever) block that has SignalEmitterComponent in prefab with the default "emit: false". The entity for this block is transient, as we can easily recreate it when needed from the prefab. This component does not have the "marker annotation". However, if someone switches the lever, the property has to be set to "emit: true". The system that does that, also has to add additional component - say LeverSwitchedCompoent that does have the "marker annotation". When the lever will be switched again, the system will have to remove the component. Is that what you mean? What is the memory footprint of storing an extra component (both in memory and on disk when chunk unloaded), is it much?

That is correct.

The memory footprint is the size of the component object, plus an entry in a TIntObjectMap (trove map data structure that is efficient for primitive types). So a maybe a dozen bytes I guess? On disk is maybe 4 bytes (an additional integer in the components list of components, but protobuf may use less than 4 bytes to represent it).

Event stuff

I guess I would probably take a slightly different approach to that damage case - instead of sending out two additional events (and I think you would agree that "OnFixedPrevention" and "OnPercentagePrevention" are kind of dodgy names) add methods for applying those types of modifiers to the damage event, which could internally retain them and determine the final result in a manner that would be predictable regardless of ordering. But I certainly agree with the principle of incremental improvements (like we have a choice anyway).

I take it then that you feel changing from a numeric priority to before/after is an improvement for when limitations have to be worked around in the meantime?

 

[Cervator]

Posting in epic thread (read the whole thing too!). Very insightful. Stickied the thread + badged Marcin Sciesinski
as architect

Also wanting to ping some people as this is a very informative thread, especially for those interested in block/entity management as well as chunk/block loading. So among others: Panserbjoern DizzyDragon Skaldarnar UberWaffe

It seems this stuff when more firmed up and tweaked in code some more would go great in the wiki, and probably in an ES article too

Thank you very much guys! Keep it up

 

[UberWaffe]

Off-Topic:

Spoiler

It seems this stuff when more firmed up and tweaked in code some more would go great in the wiki, and probably in an ES article too

Currently there doesn't seem to be some sort of high-level overview on the wiki of what capabilities exist within the engine.
Coding noobs like me have trouble figuring out what is possible (by using which classes) by glancing at the github. (Which is also rather intimidating in its scope and size for casuals like me. )
(I when there are no doc comments.)

Overview would cover no details of how it is done, just what can be done.
Details should be exclusively covered by inline code comments and Doc comments. Mainly to adhere to the Single Source of Truth best practice.

I'm willing to create a topic "Engine Capabilities" on the wiki, giving a brief overview (feature list) of what the engine can do (Blocks, Entities, Lighting, Rendering, Communication), with links to the relevant github folders/files for those who want more detail.

Will also help me get into the code as well.

 

[Marcin Sciesinski]

I'll be adding to this thread, bit by bit what we came up with. Please correct me, if you see any mistakes. This material, once approved, can go straight to the Wiki.

This diagram explains the life-cycle of a Chunk of a world. Please keep in mind, that the states (represented by rounded rectangles) are only logical constructs and as such are not represented directly in code. The arrows on the diagram represent events that are generated when a chunk transitions from state to state.

At the beginning of its life-cycle chunk does not exist. Once it becomes needed (for example player ventures into its vicinity) it's going to be generated. It transitions from "Does not exist" to "Loaded" state. Just after it is generated "onGenerated" (add actual class name here) event for the chunk is going to be triggered.

When the software has decided the chunk is no longer being needed, it is going to be unloaded from memory and stored to persistent storage (file, database, etc). It transitions from "Loaded" to "Unloaded" state. Just before the chunk is disconnected from world, an "onBeforeUnload" event is triggered for this chunk. After this event an additional event is generated "onTransientEntityBlocksDeactivated" for each block type with prefab in it, please note while strictly speaking these are not chunk events, they are very closely related with the chunk life-cycle and hence have been added to the diagram (dashed lines) for clarity.

When the software requires an already generated chunk that is not in memory at the moment, it requires to load it from persistent storage. It transitions from "Unloaded" to "Loaded" state. Just after it is loaded from storage and connected to the world an "onLoaded" event is triggered for it. Shortly after, a similar block type events are generated "onTransientEntityBlocksActivated" for each block type with prefab in it.

Notes:

• There is no plans at the moment to destroy existing chunks.
• "onTransientEntityBlocks..." events contain a collection of blocks positions, these will not include blocks that have their own specific entities stored, even if by default this block has only transient components. This might happen if a system adds non-transient component to the block entity.

 

[Marcin Sciesinski]

This diagram explains the life-cycle of a component of an entity. Please keep in mind, that the states (represented by rounded rectangles) are only logical constructs and as such are not represented directly in code. The arrows on the diagram represent events that are generated when a component of an entity transitions from state to state.

A component of an entity begins its journey when it is added to the entity. This can happen either when an entity is built from a prefab, or when it is directly added to the entity by a system. It transitions from "Not defined" to "Active". After it is added to the entity an "onAdded" event is triggered.

If a system modifies a component's properties, no state transition is made, however an "onChanged" event is called. Only new values of a component are available in the event handler.

When a chunk is unloaded, all of the non-transient entities with LocationComponent specifying its location to be in that chunk are unloaded as well. Each component of those entities transitions from "Active" to "Unloaded" state. Just before the entity is unloaded and disconnected from the world an "onBeforeDeactivate" event is triggered for each of its components. Please note, that at this point all block types also trigger "onTransientEntityBlocksDeactivated" event as well, in tandem these allow systems to handle component events for both transient and persisted entities.

When a chunk is loaded, all of its entities are also loaded and each component in those entities transitions from "Unloaded" to "Active" state. After loaded and connected to world, "onActivated" events are triggered for components of these entities.

If a all of a block entity's components are transient, the core might garbage collect it it at the end of a tick (remove it from memory). If this happens, the components of the entity transition from "Active" to "Garbage collected" state. Just before it happens, an "onBeforeDestroy" event is triggered for those components.

Whenever a block with a prefab is interacted with in any way, and it has no active entity associated with it, it is recreated from the prefab. All the components of this new entity transition from "Garbage collected" to "Active" state. After the entity is created an "onRestored" event is triggered for components of that entity. Please note, that unless a non-transient component is added to the entity, it might be garbage collected (destroyed) at the end of the tick.

If a chunk is unloaded or loaded, for each block type with prefab in it, an "onTransientEntityBlocksDeactivated" and "onTransientEntityBlocksActivated" event will be fired. These transitions are represented by the bi-directional arrow between "Garbage collected" and "Unloaded" states. More information on those events can be found in the "Chunk state diagram".

Notes:

• Only block entities can become "Garbage collected", therefore only components of those entities can become "Garbage collected".
• Only entities with LocationComponent can become unloaded, as they will be unloaded with the chunk that contains that location, therefore only components of those entities can become "Unloaded".

 

[Cervator]

Good stuff

Quick note on wiki - there is both the Xenforo (forum) Wiki and the GitHub wiki. We enabled the XF version first as a brainstorming tool, but found it sort of feature poor (it is an addon - I think it has been updated a fair bit since) and barely used it. We started using GitHub instead and have much more stuff there now and I've got some plots to auto-update parts of it with builds.

ES doc, engine (and modding) capabilities, etc, should go on GitHub for sure. I'm not sure if there'll be a use for the XF wiki, maybe with an update it could serve as the brainstorming wiki as originally intended - I figure that's more like a collaborative forum-like activity while GitHub is more for "this is how stuff works"

Either way as is classic with open source there is plenty of outdated and missing stuff to update and help in doing so would be hugely appreciated

 

[Marcin Sciesinski]

Immortius I must have blanked out, when you asked me, what would be a use case for knowing when an entity is activated but not when it is recreated (from prefab in case of transient block entities). This is actually the main use case I'd like to use in "signalling" and any other mods using "blockNetwork", I can work around it, but it's not going to be pretty (solution explained below).

In BlockNetwork class I would like to keep only network-related nodes that are in currently loaded chunks. The reason, why I don't want to have whole network (loaded and unloaded chunks) is that I would have to store it somewhere between server restarts. The only viable place would be an entity without location (to make sure it's always relevant). Storing whole network doesn't even make any sense, as I won't be able to modify components of those nodes. This might not come up with "signalling" mod, but once I start working on "electricity" where the information passed along cables is finite, I have to be able to subtract it from the storage of a producer, which might not be in loaded chunk.

Another approach could be, to instead of having a BlockNetwork at all, that groups multiple blocks into Networks and allows to retrieve some information from them (like distance between network blocks - when following cables), calculate this information each time it is needed, by traversing from one node to the other via cables, checking all neighbors, etc. This might be ok on a small scale, or for the very limited uses that are implemented at the moment, but when you consider switch in "signalling" mod, that will change its state every 1 second, checking if it's connected to other consumers every second (for example very distant ones) will eventually kill the server due to lag. Using BlockNetwork I have this information readily available (if node is in the same network as another node) and also I can cache the network distance if one is needed, clearing the cache only if this Network's topology changes, and in most cases it will not change very often, as my assumption is - state of the network (resources sent - either signal on/off in case of "signalling", electricity flow in case of "electricity") is going to change much more often, than the layout of the network.

So, now that we established that keeping state of the network in memory is beneficial, and also that we should keep only information for loaded chunks, you can see that I might be interested in receiving information when a component is activated (chunk is loaded, block with component is placed, or component is added to existing block entity) to add it to network, or deactivated to remove it from network, but wouldn't want to know, when a transient block entity is recreated, because someone looked at it, or it was interacted in any other way, or similarly destroyed at the end of the tick. If the activated/deactivated is not called for recreated/destroyed transient block entities, then OnActivated, OnDeactivated and the BlockType events called during chunk load/unload would be the only ones I'd have to use to track the loaded chunk's networks.

Alternatively, with your current design I'd have to listen to following events:
- OnAdded/OnRemoved - to track new blocks and destroyed (replaced) blocks, or when component is added/removed by a system,
- OnChunkGenerated/OnChunkLoaded/OnChunkUnloaded to iterate through all active entities for any that I have to add/remove from network,
- BlockType events to check for any transient block entities that should be/are part of the loaded network.

I don't mind doing it the way I've described, but as you can see, it's not going to be pretty (iterate through all entities in the chunk). I'm also not sure, if you even have methods to iterate through all active entities in a chunk?