The Great Convergence

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Nice! Thanks :)

Will take another look after work. Might even try to take off Monday if I can swing it, do a 4-day weekend code related marathon to test-apply the restructure to a multiplayer branch.

Oddly in my tinkering with the new GitInit task in the custom gradle-git plugin I appear to have broken something on my local system preventing JCraft's SSH stuff from working - even with the non-custom version and after clearing the Gradle cache. Still investigating when I find time.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Update: I dug around for all the remaining todo items for the restructure I could think of plus then some. Local notes getting kinda messy and I figure this is a good way to share ;)

I put them into the Nanoware org's Terasology issue tracker: https://github.com/Nanoware/Terasology/issues

Also sorted into before / after / whenever type tasks. A few remain that should really be sorted out before applying the new structure to the multiplayer branch (in its own merge branch to start with), lest we end up with a not-entirely-workable branch we now need to update manually from multiplayer just like we had to do from develop to multi ;)

https://github.com/Nanoware/Terasology/issues?milestone=1&state=open = the priority tasks - I'll be focusing on these this weekend and might even get far enough to try applying the overall restructure changes to a test multiplayer branch.

I also sorted out the weird issue I ran into while playing with gradle-git magic - it seems to be a change/bug after version 0.6.0 of that Gradle plugin. I submitted a request on the author's repo: https://github.com/ajoberstar/gradle-git/issues/31
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
If Gradle was a liquid it would be coming out of my ears after today :pinkiecrazy:

Before I've usually been able to get away with letting IDEs handy the messy details of classpaths, output dirs, assembly, archiving, and all that fun. I hope today breaks with our little "Google it, do it, forget it" mantra, I'd like for some of it to stick, the true power of Gradle seems tantalizingly close.

As for progress today has mostly been about making the game behave in the different fashions you can run it. I can do so using pure Gradle without IntelliJ now, but there are still some pathing quirks under some circumstances. Making steady progress, just out of time for tonight. Here's a random segment of notes trying to keep track of what goes where:

Fun with pathing:
  • Develop running out of IntelliJ: Terasology\out\production\Terasology
    • Is a directory, PathManager goes with installPath null and sets it to working dir. That's IntelliJ controlled, so becomes project root
  • Develop running via gradlew run: Terasology\build\mainOutput
    • Is again a directory, this time working directory is where we started Gradle, so again project root.
  • Develop running directly from distributed Terasology.jar on command line (starting in project root)
    • Is a .jar this time so we take the parent directory of the jar, which is project root
  • Develop running from distributed Terasology.exe, oddly seemed to insist using the wrong codebase (needs more review)
  • Restructure running out of IntelliJ, starts with Terasology/engine/build/classes
    • Is a directory, installPath is null, gets set to working directory of IntelliJ so project root
    • Stepping through code works fine starting with PC facade digging into engine
  • Restructure running via Gradle at first went with Terasology\engine\build\libs\engine.jar (on PCs classpath)
    • Is a file, so normally we'd go up one directory. Added hack for ../../.. which worked but is ugly
    • Example ugly path: Terasology\engine\build\libs\..\..\..\Modules\Portals\build\classes
    • "gradlew facades: PC:run" is explicit command but "gradlew run" actually still works
  • Restructure from Gradle with split out classpath to avoid engine.jar: Terasology\engine\build\classes
    • Is a directory again, we get the working directory, but this time ... it is the PC dir Terasology\facades\PC ?
    • Fixed! Assets / logback groovy stuff too. "gradlew runPC" is short for a run task in facades/PC, also a shortcut in root of plain "gradlew run"
  • TODO: Restructure running from jar
  • TODO: Restructure running from .exe
  • TODO: Restructure running as applet
  • TODO: Include testing of -homedir and outright alternate home dirs
  • TODO: Include testing of modules present in home dirs and elsewhere
  • TODO: Include source vs jar mix & match or prioritization
  • TODO: Eclipse / Netbeans?
Edit: Small update, so much Gradle and now also XML. Brain bleeding! :alien:
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Progress! Several more items down (main changes, facade changes) and I broke through a block plaguing me for a few days. Relates to how sometimes child projects in Gradle do not have tasks yet when execution starts somewhere else, under some scenarios anyway. Works now! So the new "gradlew runPC" is now seemingly flawless

After that the app distribution was pretty easy and I've got the game running out of a dist directory with one exception - module jars are copied to dist/app, not dist/app/modules, which is puzzling me a bit (snippet from the sync task below):

Code:
    into("$distsDir/app/modules") {
        rootProject.terasologyModules().each {
            from "$rootDir/modules/${it.name}/build/libs/${it.name}.jar"
        }
    }
Interestingly the complicated stuff worked right away - the right jars files are grabbed. The "into" just seems to be ignored. Maybe it is still related to how "fancy" the rest of it is. Probably not too hard to figure out, especially compared to the rest, but it is past 2 am with a workday tomorrow ;)

If you move the modules into /modules manually the game works, jar-style :)

Local Gradle stuff is just about functionally complete (short lots of polish/utility), Jenkins will use a slightly different strategy since it won't have a big fancy multi-root project, instead relying on different jobs.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Okay, first prototype Jenkins job is up - engine is pretty easy since it is in the main project: http://jenkins.movingblocks.net/job/NanoEngine/

Spent the majority of the time tinkering with dependency management for modules, including fetching the engine, the engine's standard lib jar lineup, and other modules. Got it to where I can both compile individual modules and the whole project with module dependencies not present in source form - but both still rely on placing already built artifacts in /libs manually (and at runtime binary modules aren't available in-game). That'll change, but I'm sleepy ;)

Main bulk of changes and future lineup is in modules.gradle (which is used to regenerate the build.gradle files in modules - there's a task for that now too, "gradlew refreshModuleGradle"). Plan goes something like the following:
  • Building modules in a multi-project context (dev using IntelliJ) finds dependencies in the following ways
    • If a module is present in source it'll get a hard dependency on the engine and that way get the overall project libs
    • If said module has a dependency on another module
      • Look for it in source form - if found use that via project compile dependency
      • If not then look for a file dependency binary in Terasology/libs - this is more a workaround / for testing than an main workflow
      • If still not found look up the module in binary form from our Artifactory as a normal external dependency
    • User only needs to "gradlew addModule[Name]" to get source for modules said dev cares to have in code form. Otherwise a binary will be fetched. If user does later add the source module the project will start using that instead of a binary. And vice versa if you remove a source module.
  • Building a module in single-project context (usually a module build job in Jenkins) finds dependencies ...
    • In /libs for the actual module (again mostly for testing, but could be useful for an offline dev or something?)
    • If engine is not found then fetch it from Artifactory, which should also contain a list of the transitive dependencies (engine libs) that Gradle can use
    • If engine is found in /libs ... then parse transitive dependencies out of the .jar manifest? Or just assume all the other jars will also be in /libs? Unsure if this is a scenario we really want to support as a "normal" option.
    • If module has dependencies on other modules not present in /libs then go fetch from Artifactory
That's a rough starting draft likely to change as I actually play with Gradle and especially Artifactory more. Am also moving away from modules depending on the root project (Terasology, which technically is a Java module without source, which is kinda goofy) which in term depends on the engine. Unsure if there is any advantage to depending on the root project directly.

I've also been studying further the approach the Jenkins CI project is using, which inspired this whole mess :)
For us we'll likely expand the module.txt to serve a similar purpose. While my current approach isn't too focused on versioning I suspect it would be easy to add a version identifier in module.txt, could even do so for the engine should a module want to depend on an older version (say a dev doesn't have time to make it compatible with the latest version but the module is heavily in use)

More to come.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
First module build up but I'm not entirely happy with it. As outlined above I'm trying to be flexible but inter-module dependencies are proving tricky. I got the Potions module building (although its artifact got misnamed due to the job name) by attaching the engine libs to the engine build job and copying them from there. I'm also copying in the Gradle Wrapper files via shell command.

Building the PC facade the same way should work, but while the idea of "distModeZero" theoretically works in a full multi-project workspace it breaks down in Jenkins solo module builds - I don't think the dependencies can interface with the Jenkins jobs as well as straight Gradle.

Not without Artifactory, anyway - am wondering if I should just abandon the idea of not leaving it as a hard dependency in the overall process. This is especially the case for versioned module dependencies. I can sort of hack together Portals depending on Potions by simply copying the jar from the Potions job by making the path a convention, but ...

Immortius - if you're already beginning to work on module versioning, what are your thoughts on the topic? Focus entirely on Artifactory, even from in-game / launcher when pulling modules at game run-time? If GitHub gets an API in for Releases already we could use that as a "public" source for stable modules, from within the game or from a mod site like the one Philaxx is working on, and leaving Artifactory primarily for developers. Jenkins can build stuff but it isn't quite a full-fledged versioned dependency manager.

I figure either way we'll end up using module.txt to indicate version requirements for other modules and maybe even the engine.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Small addendum from phone after going to bed. May be obvious, but wanted to see the versioning and Jenkins promotion process idea in writing:

Jenkins jobs build snapshots with job build number as identifier on GitHub change. All builds sent to Artifactory, or no? Promotion button offers parameter (type = major,minor,patch) and increments appropriately then publishes non-snapshot to Artifactory and later also GitHub releases. Can also trigger other automation like changelog and wiki regen.

Modules declare dependencies accordingly and fetches from Artifactory
 

Immortius

Lead Software Architect
Contributor
Architecture
GUI
Immortius - if you're already beginning to work on module versioning, what are your thoughts on the topic? Focus entirely on Artifactory, even from in-game / launcher when pulling modules at game run-time? If GitHub gets an API in for Releases already we could use that as a "public" source for stable modules, from within the game or from a mod site like the one Philaxx is working on, and leaving Artifactory primarily for developers. Jenkins can build stuff but it isn't quite a full-fledged versioned dependency manager.

I figure either way we'll end up using module.txt to indicate version requirements for other modules and maybe even the engine.
My primary concern initially is the use of versions to enable safe dependency resolution, identifying module incompatibility, updating modules used by saved games, and to allow you to connect to servers with different versions installed and have the right version run locally - essentially you'll be able to have multiple versions of the same module installed. This will be driven by a version field in the module.txt, and a bit of restructuring of dependencies in module.txt to also have some version bounds. Version information will also have to be part of saved games and the network handshake. And yeah, I've added a module.txt to the engine itself.

Following from this would be obtaining modules when connecting to a server, initially from the server itself and later, potentially, from a secondary source indicated by the server. I have no particular preference for what that secondary source is, although something that can be hosted as a simple directory structure would be good. Both of these will require mods to be sandboxed first, since I'm not willing to expose users to automatic downloads of potentially dangerous code.

I am inclined to leave further obtaining of modules to the launcher. This would ideally be through the launcher displaying mod site, with some smarts around handling of downloads or something like that. The mod site could potentially lean on artifactory, but it doesn't have to (and I'm not sure it adds anything).
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Oookay - late night update here :)

Have applied the restructure to a branch off multiplayer and it went very well. Amazing how little time it takes to do most of it now that I've gone through the hassle of getting it to work. Put the modules and facades in the Terasology organization on GitHub this time. Went with the two multiplayer modules from Marcin Sciesinski since they have a natural dependency and even unit tests, which I hadn't hit before. Realized we end up with a circular dependency warning in IntelliJ (PC facade compile-depends on engine, engine testCompile-depends on PC) yet neither Gradle nor IntelliJ actually fails and both unit tests and game launches normally - although I haven't finished applying code tweaks to the new branch yet, has gotten too late (4 am). So the game doesn't actually work - needs a few more hours tomorrow.

More to do tomorrow and then I have some questions and comments. For now I just wanted to at least post this update and my detailed step-by-step instructions I just tested with just one history-fail I'm pretty sure is an easy fix for the 3rd (and final) go at this - which might be really soon! Yay. Now sleepy time :sleep:


  1. Make new workspace with fresh clone, add and checkout target branch (multiplayer)
  2. gradlew idea
  3. Move applet class
    • mkdir target dir
    • git mv src/main/java/org/terasology/engine/TerasologyApplet.java applet/src/main/java/org/terasology/engine/TerasologyApplet.java
  4. Move main PC class
    • Move launchScripts into PC/launchScripts
    • mkdir PC/src/main/...
    • git mv src/main/java/org/terasology/engine/Terasology.java PC/src/main/java/org/terasology/engine/Terasology.java
  5. Move stuff into utility
    • blender_addons
    • checkstyle
    • codenarc
    • templates
      • also add VERSION from \src\template\VERSION (don't forget to tweak build.gradle later)
    • git mv protobuf utility/protobuf
    • gradle/wrapper/* -> utility/gradle-wrapper
    • rawAssets/blockShapes -> utility/blockShapes (later on move to TeraMisc)
  6. Commit checkpoint
  7. Extract Core module
    • Make a copy of the Terasology workspace (might not be needed with some Git magic, but it is easy!)
    • [for some reason I had to do "git status" in the new directory before git would consider it "clean"]
    • git filter-branch --prune-empty --subdirectory-filter mods/core multirestructure
    • cd ..
    • mkdir Core (note capital 'C' change)
    • cd Core
    • git init
    • git pull file://E:/Dev/Terasology/Git/MultiRestruct/TerasologyExtract
    • Create actual repo "Core" under "Terasology" org
    • git remote add origin git@github.com:Terasology/Core.git
    • git push -u origin master
  8. Extract blockNetwork module
    • git filter-branch --prune-empty --subdirectory-filter mods/blockNetwork multirestructure
    • etc ...
  9. Extract signalling module
    • git filter-branch --prune-empty --subdirectory-filter mods/signalling multirestructure
    • etc ...
  10. Extract Applet
    • git filter-branch --prune-empty --subdirectory-filter applet multirestructure
    • cd ..
    • mkdir FacadeApplet (to rename away from plain "applet")
    • etc ...
  11. Extract PC Facade
    • git filter-branch --prune-empty --subdirectory-filter PC multirestructure
    • cd ..
    • mkdir FacadePC
    • etc ...
    • WARNING: Something went different and Git history on Terasology.java was lost. May want to cheat and just move Terasology.java into launchScripts and move it *after* the extraction step - that'll probably retain history
  12. Delete all the extracted stuff (and for test attempts the excess modules not needed for testing)
    • git rm -r mods
    • git rm -r applet
    • git rm -r PC
  13. Commit checkpoint
  14. Move src into engine/src
    • git mv src engine/src (this can get picky, had to mkdir it and close ALL IntelliJs, not just the one that mattered)
    • Also include the old root "libs" (contains tera-ovr.jar in multiplayer - should put into Artifactory later)
  15. Commit checkpoint
  16. Update Gradle Wrapper files (from first restructure effort)
  17. Commit checkpoint
  18. Copy in Gradle build scripts (from first restructure effort)
    • engine/build.gradle
    • ide.gradle, modules.gradle, settings.gradle (overwrite), utility.gradle
    • Also copy in updated .gitignore and fletch together any diffs if necessary
    • build.gradle (overwrite) - extensive changes, but should be less next time (multirestructure --> multi)
  19. Test the shiny new Git goodness by fetching the modules and facades back into the main workspace
    • The facades got their own build.gradle files, and the modules got some .gitignores, Signalling's dependency on BlockNetwork got a case fix
  20. Deal with unit test dependencies (PC compile-depends on engine, engine testCompile-depends on PC ...)
  21. TODO: Tweak the needed Java files
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
More progress, slow and steady.

I'm probably folding the PC Facade back into the root repo, although it stays a module / sub-project just like currently. I had a couple minor reasons for keeping it separate at first but those have either been lost at this time or overridden by more pressing needs to have engine + PC Facade in the same workspace (for instance due to engine unit tests having a dependency on the PC facade)

The engine build in Jenkins thus now also attaches the PC.jar as an artifact, as well as the Gradle files and modules.gradle so module builds can simply fetch that and have the Gradle Wrapper and the "generated" safe build script handy.

Module builds now have version info including SNAPSHOT in the filename, like the BlockNetwork build. The artifact also gets published to Artifactory, but does so without good organization/module descriptors since they aren't defined anywhere / aren't deduced correctly. First try was with the Jenkins plugin for Artifactory, unsure if it'll be better to use the Gradle plugin for Artifactory instead - maybe there I can simply define "groupId" etc right in the script, which would be easy.

My next step was the module build for Signalling, since that depends on BlockNetwork. I've got the dependency hooked up in Gradle, to where it should go and resolve what I just deployed to Artifactory above ... if only there was both a group and a module name for the dependency definition:

Code:
:modules:Signalling:compileJava
 
FAILURE: Build failed with an exception.
 
* What went wrong:
Could not resolve all dependencies for configuration ':modules:Signalling:compile'.
> Could not find :BlockNetwork:*.
  Required by:
      Terasology.modules:Signalling:1.0.0
A normal definition looks like: compile group: 'org.terasology.bullet', name: 'tera-bullet', version: '1.0.0'

Tried the hacked together version: compile name: 'BlockNetwork', version: '*'

But alas, no dice! And getting kinda late tonight to play much more with it. When this is done that's a big piece of the remaining puzzle, although it doesn't include exact version promotion etc yet. Need moar tiem!
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Fixed! We can has module dependency resolution via Gradle fetching built modules from Artifactory - at build time, at least, still need to put the fetched module somewhere the game can use it too :geek:

This means I can build Signalling locally (or in Jenkins!) without having BlockNetwork in the workspace at all, because it lives in Artifactory as well. Or I can pull the source module and that'll get compiled and used. Or if for some reason I don't have internet and just want to sneakernet the binary to my laptop I can drop it in /libs and it'll be found there. In theory, anyway - not exhaustively tested yet. Like the game itself after recent changes from Immortius the Gradle magic always uses the latest version currently (from a simple " version: '+' " which means "any version after the beginning of time"), but support for picking specific module versions at compile time should be doable without too much hassle now

I also put the engine and PC facade into Artifactory while I was at it. Both now also version the built jars. Everything is named as snapshots so far, but amusingly I believe Artifactory is treating them as release builds. Might be a quirk between SemVer (the "...SNAPSHOT+123" build metadata) and Maven/Artifactory pattern detection (plain "SNAPSHOT") - not sure yet. 3 am and all that. Not even sure if we care to have the build metadata in the filename in the first place.

I've started hooking into the existing version file fun (added "engineVersion=0.1.0-SNAPSHOT+123" to versionInfo.properties - the "version" from engine) that also integrates with the Launcher. That's part of building the full zip in the new structure, the last big step for making a full game package. Although there's some smaller stuff too, like updating the actual launch scripts. Fairly pleased with the progress.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Update! I completed what I consider the main outstanding usability issue for developing with the new structure. Here's an example use case showing how easy it is to work now:
  • I'm going to try out the new stuff!
  • Currently need the right branch - until it is the new normal
    • git checkout remotes/origin/redoPCfacade
  • Always gotta add the Core module (currently, at least, might change)
    • gradlew addModuleCore (fetches from GitHub)
  • Lets hack on the Signalling module from Marcin Sciesinski ! We'll add it in source form
    • gradlew addModuleSignalling (fetches from GitHub)
  • Time to prep for IntelliJ (if running from command line can simply run "gradlew" or go straight to "gradlew run")
    • gradlew idea (preps source module for IntelliJ, fetches BlockNetwork.jar from Artifactory as it spots it as a dependency)
  • Ready to play woo!
    • gradlew run
  • Hmm, come to think of it, I do need to hack something in signalling's dependency anyway. Lets grab the source!
    • gradlew addModuleBlockNetwork (with source present the binary version from earlier will be ignored automagically)
    • gradlew idea (preps the new source module for IntelliJ)
    • gradlew run (shows the from-source version in use)
Tadaa! Special thanks to Immortius for figuring out why jars weren't loading right when running from source and generally making sense of things well at runtime :)

Works in Jenkins too, with the caveat that a dependency should be pushed and built first, so it is available when another module needs it. Pushing Signalling first without BlockNetwork available in Artifactory would fail the build.

Other comments:
  • Only one level of dependencies are fetched. If you add source module A that depends on module B it'll get fetched in binary form. If B in turn depends on C (a "transitive dependency") it will not be fetched. That's doable but also not likely to be needed for a while.
    • This might work coincidentally anyway if the dependency is already present - like a single uber-module declaring everything a dependency (even if some other modules have their own dependencies they're already prepped)
  • Versioning is hard-set to always use latest and ignored in filenames. Again doable just fine, but not super critical right away
  • The first retrieval of a dependency from Artifactory actually sets the compile-time dependency on the jar file in Gradle's local file cache, then copies the jar to /modules without version info in the file, which is then used at runtime. That probably violates some sort of best practice, but it seems minor.
  • Subsequent executions sets both compile and runtime to the jar in /modules. If it is deleted it'll simply re-resolve from Artifactory (either from local cache or remotely)
With that out of the way mainly just need a new Zip building job, which needs the launch scripts / EXEs updated. Applet too. Then we can start using it. There's more polish to do, of course.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Even more good progress today, after re-thinking approaches a time or two.

There's a working application distribution in Jenkins now, although it (on purpose) excludes all modules. I think of it more as a "PC harness" that an actual game package job would use as a base then overlay with a set of content modules. It actually runs, too, you just can't create a world as there's no Core module etc. You can fetch it as a subset of the attached artifacts

I cheated and during app distribution rename the PC jar file to "Terasology.jar" although this one just contains one whole class and a somewhat hacked together manifest that allows it to find all the lib files (which includes the engine.jar)

On top of that I think I've got an interesting approach to putting together game mode distributions. I've added the beginnings, ran out of time actually taking it to the next step with a ModeZero assembly job using it. Here's Core's module.txt:

Code:
{
    "id" : "core",
    "version" : "1.0.0",
    "displayName" : "Core Gameplay",
    "description" : "This module introduces the core Terasology mechanics",
    "optionalDependencies" : [
        {
            "id" : "BlockNetwork"
        },
        {
            "id" : "Signalling"
        }
    ]
}
The optionalDependencies (exact name?) are meant as modules relevant to this particular module's interests. I'll make Gradle smart enough to where the optionals will be fetched just like normal dependencies (source, local binary, Artifactory) although missing one probably wouldn't be a fatal error. Easy way to then use "gradlew addModuleCore" to in one go get the Core module in source and every other listed module in binary form. Then selectively override the ones you care about with source, if wanted.

Likewise in Jenkins that could be used to build a ModeZero app zip or a "ModeOmega" app zip that would have a much larger module list when we have more to play with. The Launcher could fetch the PC assembly from the engine job as the base game then offer those game mode distributions as "mod packs" or what not in its GUI. Pinging mkalb MrBarsack & Skaldarnar :)

By the way, do you guys still use the following bit from the build jobs or have the bundled version info taken over?

Code:
echo "$BUILD_NUMBER" > /home/movingblocks/httpdocs/updater/multi.ver
On top of that I envision a "strict" setting in a game mode definition (within a module, like the first round of Light & Shadow) that'll treat the optionalDependencies as an "allowable modules" list for said game mode, disabling all other modules when that's the chosen game mode. Normal dependencies will still be the only ones that'll be mandatory. After all not all modules will make sense in Light & Shadow. There'll probably be a way to override, but anyway, seems like it would be useful. What do you think, Immortius ?

That's it for now, getting late. This is exceedingly close now and I was thinking I'd do a final application to a multiplayer branch on Saturday for review and replace develop with the restructure if all appears well next day. I would appreciate any review in the meantime of the Gradle magic (can test with the line-up in the above post - still needs polish, of course) and the remaining issues listed for the restructure to see which would be absolutely needed before a merge :)

Of course - on the other hand I just realized the whole licensing / contributor agreement thing sort of went idle, and while "Apache all the things" for the stuff we've got is fairly straight forward and doesn't even take any substantial changes I imagine I've still got some doc work and other fun things I should probably get out of the way ASAP. Wee ;)
 

Immortius

Lead Software Architect
Contributor
Architecture
GUI
The optionalDependencies (exact name?) are meant as modules relevant to this particular module's interests. I'll make Gradle smart enough to where the optionals will be fetched just like normal dependencies (source, local binary, Artifactory) although missing one probably wouldn't be a fatal error. Easy way to then use "gradlew addModuleCore" to in one go get the Core module in source and every other listed module in binary form. Then selectively override the ones you care about with source, if wanted.
I think it would be better separating this out from the modules themselves. Core (or whatever) shouldn't be aware of other modules just because you want them available for development/distribution. Additionally this doesn't allow for multiple distributions involving core (like lite, standard, full distribution options). I would suggest having a separate set of files outside of the modules entirely that enumerate the modules required for a distribution/working set (distribution manifests?). I'm sure that otherwise the logic would be similar.

On top of that I envision a "strict" setting in a game mode definition (within a module, like the first round of Light & Shadow) that'll treat the optionalDependencies as an "allowable modules" list for said game mode, disabling all other modules when that's the chosen game mode. Normal dependencies will still be the only ones that'll be mandatory. After all not all modules will make sense in Light & Shadow. There'll probably be a way to override, but anyway, seems like it would be useful. What do you think, Immortius ?
What you're proposing is that gametypes have to know every mod that works with them, and have to be updated (and released) every time a new mod is created for them. This basically shuts down all modifiability, and I disagree with the approach. In truth gametypes should know nothing about their mods - mods should know what their requirements are and what gametypes they work with. If I make a new mod for ModeZero, then I should be able to release it and have it used without bothering the ModeZero author. It is up to me as a mod author to specify what my mod is compatible with gametype-wise. And partially up to the user to use the mod correctly.

The one area I had been thinking of for optionalDependencies is where a module will make use of another if available, but doesn't force it active. That is it is a compile-time dependency but optional at runtime. These act more like standard dependencies for the build system though.

That's it for now, getting late. This is exceedingly close now and I was thinking I'd do a final application to a multiplayer branch on Saturday for review and replace develop with the restructure if all appears well next day. I would appreciate any review in the meantime of the Gradle magic (can test with the line-up in the above post - still needs polish, of course) and the remaining issues listed for the restructure to see which would be absolutely needed before a merge :)

Of course - on the other hand I just realized the whole licensing / contributor agreement thing sort of went idle, and while "Apache all the things" for the stuff we've got is fairly straight forward and doesn't even take any substantial changes I imagine I've still got some doc work and other fun things I should probably get out of the way ASAP. Wee ;)
We absolutely need to get the licensing for content sorted though.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
On Core, that's just a placeholder since I've got it, sorry, I should've pointed that out :)

We'd have an actual ModeZero module, and that would likely only be responsible for defining the game type (if any). Rather than knowing of all modules that would work with it there's a known foundation set of modules that get distributed with it (if available). Only strict game types would try to limit modules, and that's the only time when it would make sense (Light & Shadow). Even so maybe modules not impacting function would be allowed, but it seems currently our modules focus on pure content unlike in MC where some are adding custom shaders etc due to the limitation of the engine. We don't have that currently although I suppose we might one day?

Alternatively, yes, we could have a "package manifest" that's similar to a game module, but I figure we might as well multipurpose if it makes sense? Really, could make a ModeZero module and just call it a manifest instead ("ManifestZero?"). Can split later if needed. Maybe what I'm thinking of are more like pre-selected packages we provide officially than outright game types, they just happen to match with game types. Other user-defined game types need not take advantage of the system, it is just an extra.

As for whether a ModeZero or game type should be updated / re-released - I actually would like to see builds run for everything relevant when there's a change, purely to catch errors and trigger notifications for involved devs. Not necessarily a re-release every time, would leave that up to promotion then auto-updates can offer individual pieces. This area is a little more hazy still though, needs more thought.

On ModeZero in particular, if we treat it as the base game with frameworks but with minimal content then in that case I do think we would control that as an official lineup. Whole thing would be Apache 2.0. Custom user modules that we don't have a lot of yet - that's more up in the air.

I'll defer to your expertise as to the specifics what we do, I just want to push forward to get something working soon :)

(All this is intended primarily as brainstorming - I'm not set on any particular path)
 

Immortius

Lead Software Architect
Contributor
Architecture
GUI
On Core, that's just a placeholder since I've got it, sorry, I should've pointed that out :)

We'd have an actual ModeZero module, and that would likely only be responsible for defining the game type (if any). Rather than knowing of all modules that would work with it there's a known foundation set of modules that get distributed with it (if available). Only strict game types would try to limit modules, and that's the only time when it would make sense (Light & Shadow). Even so maybe modules not impacting function would be allowed, but it seems currently our modules focus on pure content unlike in MC where some are adding custom shaders etc due to the limitation of the engine. We don't have that currently although I suppose we might one day?
You can add custom shaders right now, but effectively only for materials to be used by entities (you can override the core shaders, but you have to conform to their contract pretty closely). Perhaps we'll eventually get in the place where the main shaders can be overridden nicely, or post process effects introduced.

I guess the crux of my concerns is, and perhaps something that you've overlooked, is that Light & Shadow (or any gametype) may also get mods added by the wider community. ModeZero is intended as a sandbox for modding, but that doesn't mean support for mods isn't needed for other gametypes. And while some mods may only make sense for certain gametypes or in combination with certain other modules, I don't think locking down modding for some gametypes is the correct solution - I think this can be handled by mods (not modules) providing a list of gametypes they work with, or a list of modules they require a gametype to make use of.

Alternatively, yes, we could have a "package manifest" that's similar to a game module, but I figure we might as well multipurpose if it makes sense? Really, could make a ModeZero module and just call it a manifest instead ("ManifestZero?"). Can split later if needed. Maybe what I'm thinking of are more like pre-selected packages we provide officially than outright game types, they just happen to match with game types. Other user-defined game types need not take advantage of the system, it is just an extra.
I had in mind simple text file listing a bunch of modules that may be worked on or distributed together. My reasoning is that module.txt is the wrong place - these optional modules have nothing to do with the definition of the module, development of the modules, don't need to live in the module, be maintained with the module or be known by the module, or be distributed with the module. Something like:

lightAndShadow.manifest
Code:
lightAndShadow
lightAndShadowContent
core
inventory
crafting
mobs
combat
or it could be a json file with some additional settings to drive distribution.

I would also argue that "optionalDependencies" is a misnomer - these aren't dependencies really. If they must be in module.txt I would suggest renaming it to "distributeWith". As previously mentioned I am considering using "optionalDependencies" for a different purpose (similar to maven's optional dependencies).

As for whether a ModeZero or game type should be updated / re-released - I actually would like to see builds run for everything relevant when there's a change, purely to catch errors and trigger notifications for involved devs. Not necessarily a re-release every time, would leave that up to promotion then auto-updates can offer individual pieces. This area is a little more hazy still though, needs more thought.
Certainly when a dependency is changed, assuming it still falls inside the version bounds of a module, that module should be rebuilt by CI. But that isn't what I was referring to.

What I'm referring to is official releases of Terasology and its modules, and the maintaining of those modules. If Light & Shadow has a strict list of mods it allows, then if a community member Timmy develops a new mod for Light & Shadow it forces us to update the list of mods in Light & Shadow (and him to rely on us doing so), and then release the new version of Light & Shadow so people can actually use his mod. And people on older versions of Light & Shadow will have to download a new version for a one-line module.txt update.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Pile of additional work done tonight. Wanted to "finish" but spent some extra time on polish, really want to get this right. Will review the Jenkins and dist options tomorrow, taking into mind the feedback above :)

The dev side setup is easier than ever now - quick summary
  • Core module has been put back into the main repo. This avoids having to do a single step with super magic Git-Gradle in it - no addModule[id] needed just clone and run :geek:
  • When you do use addModule the overall process is a little more hardened now. What can happen is that if the Git authentication doesn't go through (like the user misses the password prompt or it times out) you end up with an incomplete module dir. That won't break the overall process anymore now and instead logs a warning.
  • IntelliJ run config is now actually working - I didn't realize gradlew cleanIdea doesn't actually delete the .iws (the workspace file, which contains the run config). So it never worked fully
  • Lots of logging cleanup so running Gradle is less overwhelming and less noisy
  • Root project is no longer a Java module, that cleans up stuff a bit
  • Checkstyle works on the command line again
  • Gradle upgraded to 1.8 - just copied off mkalb when he upgraded the launcher :D
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Another late-night update!

I'm pretty happy with the local development setup now. More polish added, logging is cleaner, and naming is more consistent. It is now "fetchModule[name]" rather than "addModule[name]" since "fetch" is IMHO less ambiguous than "add"

I've again built up a crazy fancy workspace with modules from multiple GitHub sources, multiple facades, some thing created locally others fetched from GitHub. Still works fine with local lib projects too. Runs like a charm.

Immediately next is the Jenkins magic overhaul, which should be easier now. I've got the wiki page in progress too.

However, the hour grows late, and Gandalf the Grey needs to ride to Isengard for some counsel or something ... :sleep:
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
[Edit: Wrote this last night but lost internet - posting here in the morning now]

Well, today was sort of a wash and I didn't find time to do any coding or Jenkins tweaking. Another long weekend has proven not long enough, alas!

But on the plus side I did get to the happy dev state yesterday and if anybody wants to test feel free. It is super simple now - here's a suggested lineup:
  • git clone git@github.com:Nanoware/Terasology.git
  • cd Terasology
  • gradlew run
No need to change branch, I've made the very latest version the default on GitHub. Game should run with just the "Core" module available and enabled. Round two:
  • gradlew fetchModuleSignalling
  • gradlew idea
  • Open IntelliJ and click the run button (a run config called "TerasologyPC" should show up and be in the UI)
Emphasis on "fetch" instead of "add" - nobody is going to hit that muscle memory error more than me:D

First command fetches the source for "Signalling" and the second triggers the dependency check that figures out Signalling depends on BlockNetwork which is then fetched as a binary from Artifactory. And IntelliJ is set up.

You shouldn't have to do a single other thing, and the game execution should list both BlockNetwork and Signalling as available. One time I got a compilation error that went away after doing a "Rebuild All" but I suspect that was just a quirk for me. Key goodies prepared automatically for you in IntelliJ (there are more, check ide.gradle):
  • Run config "TerasologyPC" already set up including memory settings and the -homedir parameter
  • Checkstyle hooked up
  • Git integration enabled (including for additional source modules you fetch)
  • "Automatically Make" enabled under compile settings
Bonus round:
  • gradlew fetchModuleBlockNetwork
  • gradlew idea
That swaps the BlockNetwork module from binary to source, ignoring (but not deleting) the binary jar previously fetched from Artifactory

I've got to the "fully functioning application distribution" step before and some other stuff like Javadoc is also working fine, just a question of time to polish that up and make it official.

Questions, comments, feedback, flames, cookies, and comfy pillows can be left in this thread, I'm going to sleep :sleep:
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Initial documentation is done:

https://github.com/MovingBlocks/Terasology/wiki/Codebase-Structure
https://github.com/MovingBlocks/Terasology/wiki/Project-Overview (more of a bonus)

Very much would appreciate any additional feedback, got some from Skaldarnar and Immortius already, thanks guys :)

I redid the module distribution setup and got a version working that picks up existing modules (which will always be Core solo in Jenkins) plus any included in an "extraModules" property passed in on the command line. This should allow the setup in Jenkins to work in a cleaner fashion. Able to trigger an odd compile error if I run the distApp + copyModules + pass parameter, yet take out the parameter or distApp and no problem. Probably an execution order issue. Wanting to harden against that plus make a zip then app distribution in Jenkins should be ready!

Some assorted questions:
  • Should the applet have a module.txt? Should any facades? Not really needed after PC went internal, Applet is never getting pulled from Artifactory, Android ... dunno
  • Should repos be prefixed with their type? Right now facades would be ("AppletFacade") and modules are not ("BlockNetwork"). It may not matter if they aren't mixed (modules under Terasology org, facades under MovingBlocks org) but may be confusing in user repos that have both. However, how many users are likely to create facades?
  • Should the utility dir be broken apart and some stuff put back in default locations? Would fix the Gradle Wrapper (plugin) within IntelliJ. I didn't have any solid reasoning for squishing everything in there, just seemed tidy at the moment.
  • Any comments on capitalization in module names, facades, etc? Examples
  • Would it make sense to change the author field in module.txt to a list?
More tomorrow!
 
Top