Codebase restructuring

[Cervator]

Edit: This is totally going to be called "The Grand Convergence" from now on

This is another big one where I have way too many notes and things to mention. Been mentioning it for quite some time too, including referencing the Nanoware test org on GitHub where I've extracted out a few modules and made some small facades fitting with the LibGDX way of organizing (which would probably be the way to move towards Android support). Main Terasology repo has the changes in a "restructure" branch

In short, a picture is worth a thousand words:

I've only hooked the "engine" dir into the gradlew idea so far, and it still gets treated like an old school "mods" dir so ignore the mod.txt in there - most the Gradle still needs updates

Almost everything in the old structure is now in "engine" - the idea of the LibGDX setup is to have generic engine functionality in one module and front-ends (like a PC version or an Android version) in other modules. Then you can launch one version through its specific facade without loading stuff from the others and build them separately all depending on the engine.

The remainder of the red bordered directories will be registered as IntelliJ modules automatically (soon'ish). Additionally they're all independent Git repos, which also is set up automatically for the user with commands like:

gradlew addModulePortals
gradlew fetchLibTeraBullet
gradlew addFacadeAndroid

And so on. I haven't really figured out the proper task naming to go with and there's an assortment of capitalization quirks. Most definitely a work in progress but I wanted to finally get a concrete preview into the forum. Here are a couple samples of Gradle in action:

Another one of the quirks is visible above - I've been playing with including a set of Git repos to pull automatically but 1) the authentication doesn't work unless running an exact task, the dependency-included repos just roll by without actually doing anything and 2) the dependencies run all the time even if I'm trying to run a different task. Execution flow in Gradle is ... special

The example above is several executions later and shows tracking for what things have been pulled locally. Right now that's stored in an invalid settings.gradle only consisting of one repo per line - idea is to properly do an "include 'PC', 'Applet', 'Android'" style settings.gradle that the IntelliJ module tree setup can follow properly. One tricky facet of that is that Gradle builds its project map before starting execution so a second separate execution to "gradlew idea" would be needed (unsure if you could do "gradlew addModulePortals idea" - I don't believe you can automatically depend on the idea task, but then you might want to grab several repos first anyway)

What makes the magic work are "dynamic task definitions" in Gradle where parameters are embedded within the task names themselves. Other ways to supply parameters to Gradle IMHO are too awkward. Still offering the option to do so for optional parameters like what GitHub organization/user to fetch repos from. The magic can be spotted in the new utility.gradle

So "gradlew addModulePortals" actually parses out "Portals" as the repo name (lives here)

Modules (and nothing else) are meant to go in the new Terasology organization on GitHub. There's a goofy test repo there as with AlbireoX we've taught Gooey to create GitHub repos. Not done with that yet either, need to secure and auto-create a job in Jenkins, grant user access rights, etc. Modules do not get a custom build.gradle - they all get treated the same way akin to how current "mods" are turned into IntelliJ modules.

"gradlew addFacadeAndroid" goes and grabs a Facade from this repo, which would actually go under MovingBlocks as it does get its own build.gradle that we control fully (the reason Modules do not is because any contributor could commit arbitrary commands that Gradle would happily execute while running in our Jenkins instance). The repo has "Facade" in it ("FacadeAndroid") to make it stand out vs. the other assorted repos under MovingBlocks, yet locally the directory created is actually plain "Android" - that's just a personal desire for order that might not be the best way to go about it.

Libraries can be fetched as well, I gave the command "gradlew fetchLibraryTeraBullet" a different name ("fetch") to indicate they're different beasts entirely, but again that might just be one of my random whims and properly poor naming. This is so you can tweak at something like Jitter or what not from source within the same IntelliJ project, rather than from a binary. Gradle needs a little extra magic here to either use the normal binary or the source module if it is present.

Finally, and not indicated in the top screenie, are "Extensions" - this is what I'm the least sure about as they're essentially Modules but they do get their own custom build file and may integrate more deeply with the engine. Thus we'll want to keep those in the MovingBlocks org. Examples for extensions would be hooks needed for the Leap or the Oculus Rift (both require natives and such), maybe a wrapper for Lauri's PlaTec, maybe the advanced chunk monitor window, TeraEd, and so on. Think things that aren't truly engine material, but are more custom than modules. Yet ... we might want to just cram them in the engine anyway and harden loading like begla did for the Oculus Rift natives/lib. Not sure. Immortius ?

Finally if you look closely in the module repos under Nanoware you'll see the original commit history that affected said module is intact. I've got Git commands that'll do that around here somewhere, but they're one-time-use so I'll put them in a spoiler later if somebody is git-geeky enough to want to know.

I think that's enough for now - more to share, and everything here is subject to change - feedback would very much be appreciated, I don't want to push this on the project if it isn't a good idea. Mainly it achieves:

• Extracts modules into independent Git repos so they can be treated as independent projects complete with their own version handling (for dependency management)
• Adopts the LibGDX structure offering the potential for an Android facade (I've included a demo setup I tested in a different project setup)
• Splits out some of the not very core Engine things to where they can be treated and built separately - only the overall project wrapper and the engine remain in the Terasology repo
• Cleans up the project directory nicely (bunch of stuff now lives in a utility dir)

The Module packaging closely matches the Jenkins CIplugin infrastructure - every single plugin has its own repo that can be managed automatically via IRC bot including JIRA integration and so on (most of which is still todo). That still leaves their GitHub home kinda scary (1k+ public repos) thus why I made the separate "Terasology" org purely for modules, rest stays under MovingBlocks (little unsure where the "Core" module would go)

It also very easily allows others to host Terasology modules in their own GitHub accounts while following a convention allowing for easier integration. And something something about mod repositories but it is past 2 am and I started the day already sleep deprived

Finally I have another tie-in to this whole thing related to licensing / contributing guidelines - but that's a tale for another day, maybe tomorrow

(Also, everything above is entirely based around IntelliJ, which supports multiple Git roots within one project and fun stuff like that. I haven't considered Eclipse or plain command line so far)

Edit: Oh - goal would be to apply all this right after/during the develop-multiplayer convergence event (need to research Git commit cherry picking some more ...). I've got a separate step-by-step list including prepwork that can be done early and what to pull from where. Everything would get redone from scratch when ready, nothing in the Nanoware organization is going anywhere. Naturally such massive changes would make pretty much all outstanding code in other people's repos conflict like mad, so we'd want to pull in as many active branches as possible first or have them entirely redone via copy paste afterward.

 

[Immortius]

This sounds good. Doing this just after the multiplayer merge makes sense.

On extensions... really depends on the particular case. For some things, like jitter or oculus support, building them in the engine but having them only activated if the necessary native files are available makes sense. For something like the PlateC, that should be closer to a pure module - when module sandboxing is introduced, it will need to be granted heightened privileges to access a native library, and that native library will need to be placed somewhere appropriate.

I haven't had a good look at TeraEd, but I suspect it might be a separate facade even - you launch terasology through it and it adds an extra window.

 

[Cervator]

I haven't had a good look at TeraEd, but I suspect it might be a separate facade even - you launch terasology through it and it adds an extra window.

Yep, that's true. I ran into a few scenarios where the line between facades and extensions break down. Like the idea of a touch screen interface for Android or Win8 tablets. The touch screen interface would likely be the same or at least similar between the two, but they'd have different facades - so - extension that the two facades depend on? Trying to avoid duplicating code.

Anyway, that crosses into my "over-thinking it" territory. I don't think a clear understanding of extensions and edge-case facades is really a requirement for doing the base restructure. That can come later.

 

[Immortius]

I'm actually wonder if touch screen support wouldn't just be part of the core input system, since it should align with one method of leap usage.

 

[begla]

Sounds great Cervator. Speaking of...

I'd like to restructure (and thus rewrite) all rendering architectural stuff when we've got "the great merge" out of the way, e.g. creating an abstract render system interface which can be mapped to different graphics APIs. This will also mean to remove all usages of the good ol' matrix stack and display list stuff that is still lingering in the engine at an absurd amount of places. That won't be working with OpenGL ES anyhow.

That new structure will come in quite handy there.

 

[Cervator]

Cool

Curiously begla - will that potentially lead to a new sort of repo category down the road? As in, the main engine just holds abstract calls for everything and we pair it with an actual renderer module, where we could more easily customize between a mobile module and a PC module?

That's not something I'm going to worry about right now or anything, but it stuck out as a possibility

 

[Immortius]

Sounds great Cervator. Speaking of...

I'd like to restructure (and thus rewrite) all rendering architectural stuff when we've got "the great merge" out of the way, e.g. creating an abstract render system interface which can be mapped to different graphics APIs. This will also mean to remove all usages of the good ol' matrix stack and display list stuff that is still lingering in the engine at an absurd amount of places. That won't be working with OpenGL ES anyhow.

That new structure will come in quite handy there.

I have been thinking about this too, specifically to support a NullRenderer for dedicated servers . I think the big thing will be changing the UI to use a canvasing system - having a Canvas class that provides the basic functionality (drawing textures, fonts, rendering mesh in screen space) as well as support for clipping to sub regions and transforms. Then UI elements would render through the canvas rather than the using OpenGL directly. The subregioning would allow UI elements to be drawn in terms of a space given to them by their parent element. We could also use it to support all the opacity/movement/etc animation of ui elements rather than the current render to texture method which is overkill.