The Great Convergence

[Cervator]

Name: The Great Convergence
Summary: Major overhaul of the code base
Scope: Code infratructure and supporting logistics
Current Goal: COMPLETE
Phase: Celebration
Curator: Cervator + Immortius + probably others
Related: Brainstorming thread, Github Milestone, Nanoware test org

Implementation thread for technical details and progress (and for the name, heh, heh). Mainly just making it now for links and such, more details later (use discussion thread in the meantime)

 

[Immortius]

The Great Convergence, Day 1:

Begun moving changes from develop into multiplayer.

• Updated .gitignore with restructure and cleanup from develop
• Ported improved character sound system

Along the way I dabbled in some other, related improvements:

• HorizontalCollisionEvent and VerticalCollisionEvent now implement CollisionEvent, allowing both types of events to be captured by a single event handler if desired.
• Overhauled damage system and events. Damage is now instigated by sending a DoDamageEvent against an entity. The health system will then send a BeforeDamagedEvent allowing the damage to be modified (by multipliers and scalar addition/subtraction) or to be cancelled completely. Finally the damage is applied and an OnDamagedEvent is sent. Healing works through a similar system. Additionally DamageEvents now take a DamageType, in the form of a prefab.
• Added an improvement to @ReceiveEvent methods. Previously you had to do this:

@ReceiveEvent(components = {HealthComponent.class, LocationComponent.class})
public void onEvent(SomeEvent event, EntityRef entity) {
    HealthComponent health = entity.getComponent(HealthComponent.class);
    LocationComponent location = entity.getComponent(LocationComponent.class);
    //....
}

Now you can do this:

@ReceiveEvent()
public void onEvent(SomeEvent event, EntityRef entity, HealthComponent health, LocationComponent location) {
    //....
}

That is, you can have component types as parameters of the event method, and they will both be populated with the components and used as part of the component filter. The two techniques can be mixed, with filtering encompassing both the explicit list and the component parameters.

• Added a new T randomItem(List<T>) method to FastRandom, to select a random item out of a list - a fairly common use case.

 

[Immortius]

The Great Convergence, Day 2:

Achieved a little more, mostly updating the files in the root directory with changes from develop.

Part of this was updating build.gradle. I took a little time to fix a few little bothers, and a couple of issues I discovered:

• Porting the changes to get slick2d from a maven repository, I discovered we were ending up with both the previous and current versions of lwjgl in distributions. Fixed this up (the slick2d repository is out of date though, we should consider setting our own up)
• Mod.txt is now generated from a template
• The applets index.html is now generated from a template, with the list of mods and jars automatically populated.
• Updated gradle to 1.6, as I needed to make use of a new feature.

I've also updated the checkstyle configuration, although I've made a few changes - I relaxed the style for method and class type parameters so they can be more than one letter (so ClassMetadata<CLASS> is valid, and I am also fiddling with a few other settings. Spent a little bit of time removing style violations as something simple to do.

 

[Cervator]

Stable build done last night - might be a good checkpoint for this work? Not sure how you're tracking stuff

 

[Immortius]

The Great Convergence, Day 4:

I'm lost somewhere in the new rendering code. Send sandwiches.

 

[begla]

The Great Convergence, Day 4:

I'm lost somewhere in the new rendering code. Send sandwiches.

Oh noez! *sends a couple of sandwiches and a whole lot of hugs*

 

[Immortius]

The Great Convergence, Day 9:

The core of the rendering updates have been merged. There are some rough edges (water isn't looking very good for some reason) and a few features I've skipped over for the moment.

 

[Cervator]

Bit of an update from me: Gradle magic continues in the Nanoware org's restructure branch. I am also becoming convinced that we need a "Gradle of Despair" landmark or something in-game sometime

It is a bit like our ES - takes a different kind of thinking and when you get there everything becomes pretty cool. But it sure can be quirky.

Anyway:

• Gradle project tree is put together properly now. Turns out you can't nest the "settings.gradle" files to build the tree, but you can kinda fake it and actually make it dynamic instead. Very cool.
• IntelliJ gets the right modules created, although they're not ready yet. The in-between "type dirs" are skipped so the plain "modules" directory is nothing special (to IntelliJ, anyway)
• Getting close to having sorted out what goes where as far as Gradle files, which definitions need to be made where, and so on. Just need to put the remainder of the time in before the whole thing will both compile in IntelliJ and build properly as individual modules etc in Jenkins
• Git in IntelliJ is enabled automagically for the root project, and I've prepared for including the independent Git dirs in modules (also: the Git Push screen lists everything individually, very spiffy).

Could use some XML parsing magic if somebody is up for a challenge, I'm slow as molasses at something like that

The .ipr is modified in the following way in build.gradle:

            withXml { xmlProvider ->
                // activate CheckStyle configuration with project specific rules
                def iprNode = xmlProvider.asNode()
                activateCheckstyle(iprNode)
 
                // Enable Git
                xmlProvider.node.component.find { it.@name == 'VcsDirectoryMappings' }.mapping.@vcs = 'Git'
 
                println "Stuff: " + rootProject.project(':modules').childProjects.each { child ->
                    println "Child for Git: " + child
                }
            }

The "Stuff" block outputs the sub modules correctly, but I'm not sure how to then wrangle that into the xmlProvider bit above, which currently just sets the Git mapping for the root project. Looked at the Checkstyle stuff Skaldarnar set up (or was it mkalb ? My memory is so bad) but haven't quite sorted it out yet. The actual printed output is:

Child for Git: Portals=project ':modules:Portals'
Child for Git: Potions=project ':modules:Potions'

Manually tweaked the .ipr ends up looking like so:

  <component name="VcsDirectoryMappings">
    <mapping directory="" vcs="Git" />
    <mapping directory="$PROJECT_DIR$/modules/Portals" vcs="Git" />
    <mapping directory="$PROJECT_DIR$/modules/Potions" vcs="Git" />
  </component>

 

[Cervator]

More progress - it works!

As of this commit I can run the game with everything set up purely via Git and Gradle

1. Clone https://github.com/Nanoware/Terasology
2. Checkout the restructure branch
3. gradlew addModulePortals
4. gradlew addModulePotions
5. gradlew addModuleCore
6. gradlew addFacadePC
7. gradlew idea
8. Run game from IntelliJ via Terasology\facades\PC\src\main\java\org\terasology\game\Terasology.java (make sure to include the /homedir program argument or clean out local data dir - this branch is getting kinda old)

Now naturally the goal is to have a pile of that happen on its own akin to how Gradle fetches all your normal lib jars. But I'm thrilled that it just works

I have it embedding Jitter in my local workspace as well, but that hits an issue where Jitter has its own IntelliJ project section, which fails when gradlew idea starts higher up (because in that situation Jitter counts as a module, not a top-level project). Simply making Jitter's idea {} block conditional fixes but I haven't committed that yet. Libs are kinda awkward like that and really aren't needed, but I wanted to see how much work it would be. Probably not worth the effort to really go nuts there.

Slick ends up as an unsatisfied dependency on some modules, yet the game works. Probably I've defined it funny in a spot that gets inherited.

Still a huge mess and large chunks of the Gradle stuff like building artifacts isn't working again yet. But progress is progress!

 

[Immortius]

On my end:

Making good progress on merging in changes from develop. All commits up to the beginning of June have been reviewed, and features like TeraEd, climbing and the advanced monitor have been merged. Fixed some bugs as well, both with the merge (water works now) and with the new storage system (worked around a bug in Java 7 that was breaking storage if there was a space in the path).

On the thread monitoring in particular, changed this to leverage Java 7's try with resources. When a thread is doing a task, it can do this:

try (ThreadActivity ignored = ThreadMonitor.beginThreadActivity("Eating cats")) {
   //  Do the task
}

When the task is complete the thread is marked as inactive until the next task begins. This has been integrated into the TaskMaster system though, so for the most part the TaskMaster system should be used and the thread monitoring will be taken care of. (TaskMaster provides an implementation of a task queue with multiple threads that wait for available tasks and action them).

May do something similar with the performance monitor in the future.

Also did some work getting the multiplayer branch using Moving Block's maven repository for libraries in maven central - particularly Slick (the repository we were using was not maintained and does not have the latest version) and TeraBullet.

Finally have made some good progress reducing the number of style violations - all 500+ major violations have been addressed, and work has begun on the minor validations.

 

[Cervator]

Applied the Artifactory setup to the new structure as well + organized the build files a little better. Newly added modules get a file copied in via template, and the same should happen with the yet-not-implemented create module option. The module-level build.gradle is in the new .gitignore in those repos, but of course users can edit said .gitignore with access, then add a build.gradle on the repo side. Jenkins will likely need to delete any existing build.gradle it pulls and then replace it from the template (for security reasons)

Still a little puzzled that stuff works, but then I suppose a lot of the deleted/commented out stuff relates to building artifacts, which isn't done yet.

 

[Cervator]

Getting hairy, but progress continues!

• Tested initial jar building (works for engine, modules) - was a bit surprised to see that but admittedly it is just a beginning, does not yet involve actually distributing a working app
• Updated the module gradle file with remaining content & updates from multiplayer branch - although then admittedly I messed with one of the main new bits - the module.txt stuff
• Tweaked creation & validation of the module.txt, moving it to the setup step rather than every execution
• New createModule[id] task to do that last thing I said, but need to update the Gradle-Git plugin to get an "init" task ...

I failed completely at getting the new module.txt creation working as in multiplayer branch - Gradle insisted the task was up-to-date no matter what I tried (even creating a new copy task attached to something else using different directories!). So I rearchitected instead (to a Groovy non-task approach with a more boring ID placeholder), which hopefully will be an improvement anyway (shouldn't really worry about module.txt at build time, more of a setup item) ...

Idea is that you can run "gradlew createModuleMyThing" + "gradlew idea" and get:

• New module called "MyThing" already hooked into the project tree / IntelliJ
• A "module.txt" with your module's name in it - and maybe we can deduce an author name from Git info (at least as a suggestion)
• A "build.gradle" that's regenerated every time from a template - it is in the .gitignore and mainly there for later Jenkins module builds (at which point that build file needs to be able to live independently from the rest of the project and merely depend on Terasology the project). In a nutshell this is the old subprojects {} from current build.gradle
• A initialized Git repo in your module directory and some helpful suggestions in the log about how to create a new repo on GitHub (possibly asking Gooey on IRC to create you one) and linking the two together (wondering if that can be done in Gradle too)
• All the usual asset dirs, src, etc. There is no longer a hard coded list of modules anywhere, Gradle scans for them as sub-projects in its tree (applet will still need to write out said list to the index.html)
• Everything meant to be submitted already marked for addition to Git (no initial commit yet)

Another thing I've been pondering lately is where to define which modules should be fetched automatically in some convenient "get me all the standard modules please" command. If we do the "Module Zero" thing (it's looking good after some IRC discussion Immortius - thanks for the name, hehe) maybe MZ modules could live in a grouping on GitHub like another Organization

• MovingBlocks - engine, facades, libs, etc
• TerasologyCore - "Core" module and all other modules part of MZ. Everything here could be under Apache 2.0 as it is almost like the extended engine (a raw game mode without a specific goal / theme)
• TerasologyMods - Community modules in use in actual game modes like Light & Shadow. Follows the licensing/guideline approach I posted about. Yes, name is likely bad, I'm sleepy

Of course, I could just be overcomplicating things as usual. I'm pretty sure I'm not becoming a Gradle wizard, much more likely a Gradle charlatan with plenty of stuff needing tweaks before it is truly viable. Technically this whole approach isn't fully proven yet, but so far I haven't hit any unsurmountable obstacles.

 

[Immortius]

Except it was "Mode Zero", not module.

Have you considered using dependency information from module.txt to do recursive fetches?

This is probably off topic but some feature in my personal roadmap after GC are:

* Overhaul block definitions to be more like prefabs, with a lot of block features moved into components.
* Support for entity creation during chunk generation.
* Mod support proper. The current module system is the low-level basis, but support for gametypes, mods, world generation and mods proper is needed. Various improvements to modules too.
* Module sandboxing and auto download from server.
* An extension mechanism whereby a module can add extra components to blocks and prefabs defined in other modules.
* Icons as assets.
* Asset reloading at runtime...

Actually, I'll move this to another post and ask for feedback on priority.

 

[Cervator]

Oh, sorry, Mode Zero was what I meant

I need to test and re-enable the module dependency parsing sometime soon. You mean using a Mode Zero module (defining the Mode, wee) with all the related modules as dependencies, fetching each via Git? That sounds interesting

 

[Immortius]

Got a little further - integrated some of world generator mod support, liquid tint, fixed a couple of rendering bugs. I've chosen not to integrate the perlin with setup generator for the moment due to concerns over the settings not being stored in the world - so creating a new world will mess up existing worlds. Made a note to address this in the next modding arc (probably immediately after the merge is complete).

Did spend a little time cleaning up the module support, making sure they are called modules in the code and changing mod.txt to module.txt. The engine module is now treated more like other modules which removed some duplicate code.

Also added begla's particle effects, although I used prefabs to define them rather than a helper class. Took advantage of the damage type system to drive when different particle effects would play.

 

[Cervator]

Back to work a bit this weekend. Did some tweaking here and there and submitted a PR for the Gradle Git plugin to introduce a "GitInit" task we'd use during module creation to prepare a new module for Git integration. Just barely got it working enough for testing - maybe later round two can be setting up a remote repo on GitHub automagically

Either the jar I built and put into our Artifactory isn't quite right, or something cache-y is messed up on my local system as some of the plugin's dependencies aren't met. The .pom doesn't download for me right now, although at first I did get a few dependencies and was able to test the init. So not checking in the pending update I've got locally that uses the new GitInit just yet.

 

[Immortius]

Finished merging! Unless there are any engine features I outright missed. Final task was getting the player's light fading when they select and deselect a torch, and have this working in multiplayer. Just to make things interesting I had changed this so the light component is only added when needed and removed when there is no light, so the rendering system wouldn't have to skip over players with no torch. Fading is done through an additional component that specifies the target values and removes itself when done.

I managed to get the check style violations down to zero, and then merged in mkalb's additions - bringing the number of violations back up to 1,500. Some of these were false positives (it was reporting errors that conflicted with IntelliJ behavior and good sense), but the remainder are legit.

 

[Cervator]

Latest multi works fine for me if I run from source - but oddly the build from Jenkins fails to launch? This is after thoroughly cleaning out old data dirs, trying to launch via the jar rather than exe, and even trying from source without -homedir to rule out the use of a broken data dir I haven't found (worked fine).

Weird? Anybody else?

Edit: Oh, might help if I include some more details:

jar:file:/E:/Dev/Terasology/Builds/multi92/Terasology.jar!/
 
15:28:02.739 [main] INFO  org.reflections.Reflections - Reflections took 331 ms to scan 2 urls, producing 302 keys and 1424 values
15:28:02.755 [main] ERROR org.terasology.engine.Terasology - Uncaught Exception
java.lang.NullPointerException: null
    at org.terasology.world.generator.WorldGeneratorManager.refresh(WorldGeneratorManager.java:47) ~[Terasology.jar:92, multiplayer, 2013-08-18_10-30-27, pre-alpha]
    at org.terasology.world.generator.WorldGeneratorManager.<init>(WorldGeneratorManager.java:39) ~[Terasology.jar:92, multiplayer, 2013-08-18_10-30-27, pre-alpha]
    at org.terasology.engine.TerasologyEngine.initManagers(TerasologyEngine.java:445) ~[Terasology.jar:92, multiplayer, 2013-08-18_10-30-27, pre-alpha]
    at org.terasology.engine.TerasologyEngine.init(TerasologyEngine.java:148) ~[Terasology.jar:92, multiplayer, 2013-08-18_10-30-27, pre-alpha]
    at org.terasology.engine.Terasology.main(Terasology.java:54) ~[Terasology.jar:92, multiplayer, 2013-08-18_10-30-27, pre-alpha]

Game then crashes, never makes it to the main menu. The line from WorldGeneratorManager is:

for (Class<?> generatorClass : module.getReflections().getTypesAnnotatedWith(RegisterWorldGenerator.class)) {

So maybe the game when launched from the build fails to find any RegisterWorldGenerator classes and crashes? Or it finds nothing with module.getReflections? A quick null check added there might help diagnose / log nicer error - but I've gotta go for a bit

Running with Java 1.7.0_07-b11

 

[Immortius]

I've probably broken something when the engine is a jar, will investigate.

 

[Immortius]

Ok, fixed. Addressed a few checkstyle violations as well, down to 140 or so.