depth of a file, "internal" folder

[manu3d]

I'm noticing a pattern where some seemingly unrelated-to-each-other files (usually classes and interfaces) are left at a higher level in a folder while further subfolders exist alongside them. I.e. look at the entitySystem folder, which contains many subfolders but also some "loose" interfaces. Even its subfolders mimic this pattern and loose classes and interfaces can be found at the same level with deeper subfolders, i.e. look at the entity, event and systems folders.

Should these files thought of as "in common", used in multiple places by code located in the subfolders? Or should they be thought as somehow "more important" than the code in the subfolders? Or should they be considered "more readily available to other portion of the software"?

Furthermore, "internal" subfolders often crop up, i.e. in the entity, event and systems folders mentioned above, but also in the engine folder. Can somebody characterize for me the "typical" content of a generic "internal" folder?

 

[Cervator]

Definitely a question for Immortius - ping!

 

[Immortius]

Our top two packages are org.terasology, which is just to ensure we avoid clashes with any other library - general advice is to use a domain under your control for this (like movingblocks.net or google.com) but reversed, since these are globally unique. Technically modules in the broader community should use a separate root set of packages to avoid potential conflicts with other packages - I believe they are all using the module name as an extra level below org.terasology at least, which serves this purpose.

After that packages split code by purpose, so we have packages for asset handling, entity system, persistence and so forth. API classes (those that are intended to used and developed against), particularly entry point API occur in the top level packages, implementations occur deeper down. Typically I like to keep a division between interfaces (the contract that is exposed) against implementation (how the contract is fulfilled). I've been using "internal" packages to hold implementation classes where there is only one implementation - where there are multiple implementation I have a named package for each implementation describing the nature of that implementation (e.g. "null", "openAL", "windowAudio" audio system implementations).

This pattern continues down the hierarchy, so org.terasology.entitySystem has subpackages for covering code that deals with entities, events, prefabs and systems. I'm not sure why component interfaces are at the top level - they could be under entity or a components package - but it isn't really worth moving them.

The division of API vs non-API classes is then made concrete through the use of the @API annotation on either packages (via package-info.java) or classes. Only those classes/packages with the @API annotation are available to modules (plus any packages/classes from libraries we manually make available).

 

[manu3d]

Thanks Immortius for your answer.

It's interesting that you talk about packages rather than folders. At this stage, no doubt because my limited knowledge of the tools, I mostly think in terms of files and folders as seen through the project view in IntelliJ, hence my question. To find where to even begin looking at the code was somewhat non trivial and I had collate pieces of information from the wiki to figure out that on Windows an exe in the facade launches Terasology.java which in turns launches TerasologyEngine.java which in turn launches all sorts of subsystems and managers and is the where all the fun begins. Except that it is Terasology.java that provides the subsystems list (managers however are not subsystems!) and even set the first game state. Which it all makes perfect sense... eventually!

I've been wondering in this context if there is a way to add a description to folders. This would allow for a non-obviously named folder (i.e. internal) to describe its content. Or perhaps more commonly, for a higher level folder to describe its less obvious content or the meaning of a known word in Terasology's context. I.e., it's obvious that a network folder is about networking. But what does the word "logic" means -in the context of Terasology-? And what is "monitoring" about, what is actually monitored and when? And what's the registry? If a folder had a clear "most important" file in it perhaps this wouldn't be necessary. But upon entering an engine folder one is greeted by a number of subfolders and a number of interfaces and sometimes classes usually without a clear ranking to help in understanding what to look at first. Another way this would be unnecessary or of limited use is if the top-level interfaces and classes all had -one-, verbose(ish) introductory comment detailing what they are for and what context in the software they have been designed for or tend to be used for. I.e. if you look at the files in the engine/entitySystem folder and systems folder, some of the interfaces have comments for their method, but not one attempts to explain what those interfaces are for and in what context to use them.

Note: I realize these are things that eventually one can figure out going through the code. But I'd like to understand if this is just my problem, i.e. because there are ways to obtain this mid-level information somewhere else or because after a while everybody gets acquainted with the code and there is no need for explicit mid-level information.

 

[Cervator]

You've pretty much just described a Javadoc package descriptor, which I agree would be a huge help to have more of. We actually have quite a few of them, but unfortunately mostly they're just used to tag entire packages for the API, and lack any actual documentation

There are a few good examples, like https://github.com/MovingBlocks/Terasology/blob/develop/engine/src/main/java/org/terasology/audio/events/package-info.java

Likewise often javadoc headers end up just having auto-generated stuff. We really could do a better job there.

Wiki introductions are good too, but should probably remain high level and refer to the javadoc as it is easier to keep that up to date, since it is right there in the code.

It may be a good contributor friendly task to go hunting for these kinds of situations, learn or ask about how something works, then submit javadoc PRs at which point any remaining inaccuracies and possible improvements can be discussed. Then suddenly we have, gasp, documentation!

Edit: As for the wiki, have you seen https://github.com/MovingBlocks/Terasology/wiki/Codebase-Structure ? That might be another good spot to make improvements, I didn't quite finish it (story of our collective lives)

 

[manu3d]

Thank you Cervator for your reply! I am glad there is already a mechanism available. As I "drill down" from the facade through the TerasologyEngine to deeper code (i.e. the Rendering System) I will certainly look out for opportunities to either add meaningful package-info.java files or appropriately edit existing ones. As my "code spelunking" activity is mostly driven by the specific goal of implementing a sky system, I won't be going too much "sideways" into things that are unlikely to be related to it. Besides, there are things such as networking, security or even audio that I have absolutely no knowledge of and I'd be hardly qualified to even glance at them! But I will certainly help where I can.

Concerning the codebase structure page, yes I am aware of it and it is what eventually made me understand where to look. For my (absolutely subjective) way of reading it the information in it is quite important and useful but the order of the sections reflects the pre-existing knowledge of the author (you!). If I wanted to somehow "lead by the hand" a new contributor into the project I would talk about the facade first, as it is the point of entry in the software. Then I'd talk about the engine, followed by the modules and the "other files" section. Only then I'd mention gradle, which, important as it is, is an auxiliary tool: it's the grill, but not the meat. The last two sections, about legacy and common issues can stay as they are or be swapped, as the latter is about present, more relevant issues and the former will tend to be less and less relevant as legacy contributors will be eventually outnumbered by contributors who joined after the great convergence. But I'm being fastidious and I better stop it here.

Ultimately I think this discussion will be rendered largely obsolete by the publishing of a mature, stable, engine API, which obviously will take time. But there will always be somebody who wants to jump in and add features to the engine itself. Making sure the source is well documented and have clear "paths" into it, to learn it efficiently, will be very useful to this kind of people.

 

[Immortius]

It's interesting that you talk about packages rather than folders.

Packages is the Java terminology for the source code "subfolders".

At this stage, no doubt because my limited knowledge of the tools, I mostly think in terms of files and folders as seen through the project view in IntelliJ, hence my question. To find where to even begin looking at the code was somewhat non trivial and I had collate pieces of information from the wiki to figure out that on Windows an exe in the facade launches Terasology.java which in turns launches TerasologyEngine.java which in turn launches all sorts of subsystems and managers and is the where all the fun begins. Except that it is Terasology.java that provides the subsystems list (managers however are not subsystems!) and even set the first game state. Which it all makes perfect sense... eventually!

Probably the weakness here is that Terasology is open source and exposes that information - it is pretty much irrelevant for the majority of users. If you were picking up Unity for the first time, you wouldn't start by trawling through how it starts up or all the subsystems plug in - that code isn't even exposed. The important things are around the entities and assets, and creating a module.

Ultimately I think this discussion will be rendered largely obsolete by the publishing of a mature, stable, engine API, which obviously will take time. But there will always be somebody who wants to jump in and add features to the engine itself. Making sure the source is well documented and have clear "paths" into it, to learn it efficiently, will be very useful to this kind of people.

I hope as things progress there will be less and less jumping in and adding features to the engine. It is intended that people be able to create modules to do what they want to do, rather than having to modify the engine for everything. A lot of the things in the "logic" package should ultimately be moved into modules, for example.

It is certainly true that we need better api documentation - both javadoc and general reference documentation. However, time is expensive so such efforts need to be directed, and prioritized against development.

 

[manu3d]

Probably the weakness here is that Terasology is open source and exposes that information - it is pretty much irrelevant for the majority of users. If you were picking up Unity for the first time, you wouldn't start by trawling through how it starts up or all the subsystems plug in - that code isn't even exposed. The important things are around the entities and assets, and creating a module.

True. In the context of the sky system and the discussion we had about it a couple of missing engine features were multiple cameras and how to use or modify the existing rendering system to handle/render them. As it isn't obvious for me, looking at the project structure, how rendering/camera code interacts with the rest of the software, I simply picked the strategy of starting from the beginning and finding where rendering comes into play and how. I could of course wait for you, begla or other experienced people to implement everything necessary and then proceed to implement the sky's systems and components, interacting with engine only from a module's perspective. But I'm aware these features are not exactly priority, and I've never had the chance to study a codebase this size, so, why wait when I can learn something new and perhaps help along the way?

It is certainly true that we need better api documentation - both javadoc and general reference documentation. However, time is expensive so such efforts need to be directed, and prioritized against development.

Ah, naive and inexperienced me! I thought documenting was an inextricable part of development!

 

[Cervator]

manu3d - ran across this by chance, thought you might like a link

http://forum.movingblocks.net/threads/block-entity-behavior.792

It is another example of amazing documentation we have - yet probably nobody knows/remembers that it exists because it is lost deep in the forum somewhere

As usual whether it is actually accurate as per current state is another question, but it has neato diagrams and stuff! Might be good to add some of it to https://github.com/MovingBlocks/Terasology/wiki/Entity-System-Architecture sometime

 

[Immortius]

There is a fair bit of doco in the various incubator threads, that needs to be extracted out and cleaned up.

 

[Mike Kienenberger]

If I wanted to somehow "lead by the hand" a new contributor into the project I would talk about the facade first, as it is the point of entry in the software. Then I'd talk about the engine, followed by the modules and the "other files" section. Only then I'd mention gradle, which, important as it is, is an auxiliary tool: it's the grill, but not the meat.

For most new contributors (and this should continue to become more true over time), knowledge of the facade and most of the engine is the least important part. Knowledge on how to set up modules, add assets to them, write NUI interface code, work with the entity systems, and work with specific public engine systems (EntityManager, LocalPlayer, BlockManager, Time, NUIManager, BlockSelection, BlockEntityRegistry, WorldProvider, world generation, particles are the ones that come to mind), and work with some of the core modules (Inventory, etc) is all that is needed or will ever be needed.

It would be an interesting survey to take our active module list and compile what systems and events each one use and to what extent. That would would likely give a good picture of what's the most important entry points into Terasology. I've tried to list things above in the order that I found most used, and probably the most important to learn first based on what work I've done in modules.

I should probably make this into a wiki page when I get time.

 

[manu3d]

For most new contributors (and this should continue to become more true over time), knowledge of the facade and most of the engine is the least important part.

I can see I've heated up things a little. I'd like to apologize about it. I only meant to share the difficulties I'm experiencing and how I'm addressing them.

In fact I mostly agree with you Mike. If a new contributor wishes to develop or expand a module and is possible to do so leveraging the existing feature set of the engine I can see how there's no reason to look much further than the systems you mention (indeed it would be nice if your list and some associated commentary made its way to the wiki, so that the new contributor might know about it!). In this context, my suggestion of placing the facade first in the codebase structure page does not imply -importance-. The facade can be mentioned briefly, as the simple component it is, swiftly progressing to describing the engine. But (for me) it is a logical place to start because that's what starts the engine.

New contributors of the type I mentioned at the beginning of the previous paragraph can proceed to the engine section and your list of systems. New contributors who need to expand the engine for whatever reason must first find how the engine works, at least in broad strokes. To do that I found it useful to use the facade literally as a main door. I didn't stop long to look at it: I just used it to enter the Terasology building's "lobby" (the TerasologyEngine). From there it was relatively easy to obtain a map of the whole building and proceed to the floor and rooms I'm interested in. But I needed to get to that lobby and from the project view or the Codebase Structure page its location isn't obvious. Also, to make a specific example, the engine's code mentions subsystems, but it is the facade that provides them. So, even if I realized TerasologyEngine.java was the lobby, I still needed to take a look at the facade to understand what those subsystems are. In short, for the second type of contributor the facade is not overly important, but it has some important pieces of the puzzle to offer and is a good starting point to get a sense of the context the engine works in.

 

[Cervator]

Oh I don't see any heat, something a well-placed smiley probably could have made clearer

Speaking of making things clearer: We need diagrams! IMHO that would probably be perfect way to get the "flow" of the architecture real quick