Implementation gestalt-entity-system

[Immortius]

Name: gestalt-entity-system
Summary: The entity system from terasology with improvements and cleanup
Scope: Engine
Current Goal: Pondering
Phase: Early planning
Curator: Immortius

Some thoughts:

• Switch components to be interfaces, and generate their implementations
• Improve safety around component usage - defensive copying
  
• Attempt in-memory database support for querying and possibly transactions
• Multithreading support w/ atomic short lived actions (this is used in a few systems such as DGD, the idea being to have short actions that make a change, and if they change any objects that have been changed in the meantime on a different thread then the action is repeated from the beginning).
  
• https://github.com/buschmais/extended-objects
• Querying?
• Java 8
• Sending events to Prefab
• Multi-entity prefabs
• Improvements to entity ownership/composed of vs references

 

[manu3d]

DGD as in http://en.wikipedia.org/wiki/Dworkin's_Game_Driver?

 

[Immortius]

Yes, that's the one.

 

[Immortius]

Another thought - changing T EntityRef.getComponent(Class<T>) to Optional<T> EntityRef.getComponent(Class<T>).
Also generally consider where lambdas can be take advantage of.

 

[Immortius]

I will be starting on this shortly, beginning with some design, so will be looking for feedback and suggestions. Probably the big thing is coming up with a thread-safe approach so that the entity system can be safely interacted with by background tasks.

 

[Josharias]

• Switch components to be interfaces, and generate their implementations

I will likely get some purist flack for this... I do enjoy the flexibility of being able to mix in some code alongside component fields. In my opinion, putting logic closer to the actual data is useful from a maintainability/understandability point of view. It has felt odd particularly for the inventory system having to go to the InventoryManager for some things, which completely abstracts away the component side, but then for events and inventory configuration one has to deal with the component directly. Having basic manipulation of the component at the component (this may not always be possible with the parts that orchestrate across entities) could help new module developers to quickly understand how to use a particular feature.

Additionally, allowing systems to be extended by having components share a common interface has also been a convenient tool for extending functionality in a child module. This is used extensively in the Workstation module to allow extension of the process system with new process parts that all share a common interface, but provide their own implementations. For example, the ManualLabor module provides heat process parts that allow the process to require another entity on the heat network to be producing heat. This heat process part can be added to the processes prefab as an additional component, when the Workstation module iterates over all the processes components it will use all components with the shared interface in the process.

I am sure there are other ways that extension could be implemented, but I have found it to be a nice flexible system in its current form.

 

[Immortius]

Stopping that misuse is one of the reasons I was considering changing to pure interfaces in the first place . The other being the injection of the code needed to support some of the planned improvements.

I don't disagree with having some data logic in the components (and perhaps this could be done though Java 8 default methods). But there shouldn't be any game logic in them. The benefits of having game logic moved into systems is both support for batching updates, event handling and allowing the system to be extended or replaced to change behavior without having the change the data model.

I would also say you are only seeing benefit in extension through shared interfaces *because* you have started adding game logic into components.

This is used extensively in the Workstation module to allow extension of the process system with new process parts that all share a common interface, but provide their own implementations. For example, the ManualLabor module provides heat process parts that allow the process to require another entity on the heat network to be producing heat. This heat process part can be added to the processes prefab as an additional component, when the Workstation module iterates over all the processes components it will use all components with the shared interface in the process.

An entity system is all about decomposing behaviors into components that are then composited together to form entities. If you have inheritance, it means you haven't decomposed a behavior. In this example, you haven't decomposed the behavior of being a part separate from the input requirements of a part.

I would suggest in the first instance that those parts should be independent entities, with a PartComponent and then other components for input requirements/outputs behavior. Perhaps even inputs and outputs could be made additional entities. The lack of support for entity structures is one particular weakness of the current implementation (particularly in prefabs) that needs to be addressed. It needs to be much easier for them to be taken advantage of because it helps specifically in these sorts of situations.

In essence I feel you have strayed from the entity system model to a "component system" model. And by doing so you have lost the extensibility benefits the entity system model brings, so have had to use inheritance to provide them instead.

 

[Immortius]

I guess I would also say I am wedded to exploring entity systems rather than component system (which is the vague unofficial terms to distinguish between having logic in separate systems or embedded in the components). It may turn out that we never end up using gestalt-entity-system in terasology, or it may end up in some sort of Terasology 2 project, we'll see.

Anyhow, establishing some scope for gestalt-entity-system:

In Scope:

• Components
• Entities and management thereof
• Prefabs
• Event System
• Lifecycle events
• Persistence
• Networking support, but not actual implementation

 

[Josharias]

I admit, I thought "entity system" and "component system" were all the same. Apparently there are some nuances that my web searching skills did not find. I will look up some reference material about the differences between. Do you have some articles to share that would assist aligning with your vision of the entity system?

Question, do you still condone using prefabs as a way to store static game data? This is basically what is happening in parts of the Workstation module, where we use a prefab not to create an entity, but to grab data. Would it be better to make different asset types to hold this data instead of prefabs?

In the meantime, I will work on getting the modules I actively develop in a more compatible state for Entity System vNext. Please message me if you see something else that does not align with the Entity System vision so I don't have to do things twice.

 

[Immortius]

To be fair the particular nuances between "entity system" and "component system" is only described in T-Machine's articles (see http://t-machine.org/index.php/2007/09/03/entity-systems-are-the-future-of-mmog-development-part-1/ ). However the distinction is real and has some clear pros and cons regardless of the names used. I certainly still feel the core of what T-Machine describes is important. You may also want to consult https://github.com/MovingBlocks/Terasology/wiki/Entity-System-Architecture

I will make sure to post requirement and design musing here as I get to work on things, which should help clarify what I'm thinking.

On prefabs to store static game data - depends. I still think having references to prefabs are a great way to save bandwidth in multiplayer - if I can say an attack does engine:fireDamage then that could be communicated with a single integer rather than a having to send across all sorts of information. But the same is true for for assets. I guess it depends on the complexity of what is being described. I still think prefabs are fine to describe some things. Either way works though.

 

[Immortius]

Some thoughts, starting with thread safety since that is probably the most demanding feature:

• Entities will be similar to now - just an id that is used to group components and reference the component, wrapped in a convenience class.
• Components will hold data.
• For thread safety, entities and components will be accessed in a transactional context. When accessed a copy will be made of the entity/component. When the transaction ends an attempt will be made to save any changes.
• Each component will have a revision number, that will be incremented when it is saved. If a transaction ends and a modified component's revision doesn't match the current revision, the transaction is rolled back (with the expectation of being re-run). Basically, optimistic concurrency. This obviously suits small, short-lived transactions that touch on few entities - in the preferred case of few conflicts this allows a lot of parallel processing.
• For events, by default they will be asynchronous - the current thread will not block waiting for them. These events will be queued up until the successful completion of the current transaction, and then will each be run as independent transactions. Optionally events may be synchronous, in which case they fall within the current transactional context. Having independent events helps keep the transactional contexts short-lived though.

Obviously there is a lot of standard db concepts here so worth investigation of possible existing technologies to leverage. Something like Neo4J or OrientDB may be usable, but a lot of the nicer NoSQL dbs don't have performant transaction support. Leveraging an in-memory db would allow indexes and more advanced queries though.

Component definitions as interfaces comes into play here, as this would allow the implementation to add whatever features we desired - lazy copying of attributes, additional info like revision numbers, etc.

 

[Immortius]

More requirement thoughts:

• "Naked" entities (entities without components) will not be allowed. An entity that has all of its components removed will be destroyed.
• There will be a PersistenceInfo component that provides information important for persisting an entity - prefab, persistence parent, etc.

 

[Immortius]

• There will be two ways of interacting with the entity system - atomically, and transactionally. If no formal transaction is active, then essentially each interaction will be a transaction. (e.g. If you create an entity as part of a transaction, it will be held as part of that transaction and all actions that are part of the transaction will succeed or fail together. If you create an entity outside of a transaction, essentially a transaction will be created and committed just for that action).
• Both types of interactions will be thread safe, although working atomically is more at risk of lost update style issues.
  
• Adding a component to an entity that already has that component will fail (currently with an IllegalStateException, but may need some thought. Primarily this will be an issue with competing threads working on the same entity).
• Transactions will be thread local. There will only be one transaction at a time on a given thread. Need to think about the effects of requesting a second transaction, but probably should just join the existing transaction, and cause a commit request to be ignored (e.g. if you begin three nested transactions, three commits would be needed to end the transaction).
• Entities/components will be cached within a transaction, so if the same component is requested repeatedly the same object will be returned.

 

[Immortius]

After a long break... I've committed an initial step with the core of an entity system, with atomic operation support. This also includes generating component implementations from interfaces. This isn't really useful at this point, but from here I will establish benchmarking so that I can measure the cost of different implementations.

During my break I've been thinking about the qualities I would like the entity system to have. Most of the previously mentioned goals are still goals:

• Making the entity system usable outside of Terasology
  
• Thread safety
• Atomic operations
• Transactions
• Investigating use of in memory db, support for indexes or otherwise new ways to quickly find desired entities
• Improved safety when using components

But there is one more important goal I wish to add

• Minimise object allocation/garbage collection

Essentially, all components and the data types they concern could be obtained through object pools. When a component is no longer in use it would be returned to the object pool, to be reused. This would allow memory to be managed with much more care - objects would only be deallocated when a pool is purged. To support this components are now created via the EntityManager - this can then be updated to use object pools. A releaseComponent method may also be introduced to return components to the pool, in situations where this cannot be managed automatically (typically, outside of a transaction).

Related to this, for the moment I've removed EntityRef from the entity system - I want to re-explore how these will work. The only method of having a single entity ref for each entity, and marking them as disposed as needed seems burdensome in a mulit-thread situation, and having methods on the entity ref doesn't sit will with having two context in which an entity could be worked with (atomic or transactional). They will probably re-emerge in some form, but not like were.

 

[Cervator]

Good to see some movement again

Only thing I'd add is a link to the sector thread I wrote not too long ago, acknowledging first and foremost that I'm not an expert on the topic. It is a bit of a read but the relevant part comes down to the idea of having isolated groupings of entities. Different worlds don't really need to know about all the entities in the other and such. But even within a (large) world it might make sense to eventually keep some sub-groupings. I get into examples from EVE Online a bit, where you can isolate constellations (groups of solar systems) and dynamically apply more machine power to them (reinforced nodes).

I don't know if that's a good idea or if a really fast DB approach would be better, but in either case it isn't likely to be an immediate consideration/need. I wanted to bring it up just for awareness maybe for another round down the road (engine v3.0.0?)

 

[Immortius]

Hmm, I guess I'll have a think about it. Essentially it is just support for having groups of entities that are loaded and unloaded together, and to iterate entities within a group or set of groups - it would be up to the using application to decide what these groups are. And there needs to be some support for moving entities between groups. For multi-machine hosts, you would probably have a machine per sector and move entities between them as necessary - more of a networking thing, and entity ids probably wouldn't carry across between machines.. I'm thinking guild wars 1 style architecture with character data that moved between instances and even between data centers seamlessly.

 

[Cervator]

Yep, that kind of stuff. Trickiest bit is likely the overlapping areas, like if combat between players is happening right on a sector edge. I threw some ideas into the thread but am not sure how best to implement an "affinity" system of sorts that'll bias a set of related entities to stick to one sector side if needed or instead dynamically move the line somehow.

Destination Sol might again be a good secondary use case to consider. There we might one day boost the galaxy size up massively to where you could again have whole parts of the galaxy not needing to know about the rest, yet end up with border systems of some sort. Although maybe there you can cheat and use chasms of vast open space between systems to encourage players to only jump via gates, giving some border options to switch between groups easily.

Certainly some of all that is in the implementing app rather than Gestalt. Unsure where the ideal breaking point is, how much possible utility could be shared.

Great to have you back btw!

 

[Josharias]

In perusing through the physics system, sectors comes to mind as a way to parallelize physics by having separate physics engines for each sector. This also relates to groups of entities in that one could create dynamic groups of entities in islands of connected active chunks. There could then be a dedicated physics instance for that group of entities. And when the groups got close, you could merge them together.

But then taking groups of entities further, this could also form the basis of multi-world/dimension stuff too if each chunk was an entity that was parented to a single world entity.

Not sure how much of that relates to your current line of thought, Immortius, but would be interesting to be able to support some of those concepts.

 

[Immortius]

At this stage I'm not thinking of actually going back to Terasology to implement anything, but I feel that ideally it would go the way of everything being an entity, including worlds and chunks. Certainly those considerations are going into the entity system design. What you describe in terms of chunks being parented to worlds was also what I was thinking.

Dynamic partitioning of the physics world seems like a good approach - I imagine this could be done internal to the physics engine though, doesn't really need to touch on the entity system.

 

[Josharias]

Immortius, recently I pulled out the inventory system from the engine into an external module. In order to avoid sandboxing problems, I had to stop using annotations to provide extra metadata of how a component could be used (in this case @ItemDifferentiating) and instead decorated the components with an interface. You have mentioned doing some more code generation around components in order to accommodate transactions and such, and using interfaces like this could get in the way of how you want to define components. Do you figure this kind of mechanism will work with the direction you are thinking to take with the entity system?