Project Logistics Arc

Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Name: Logistics Arc

Summary: Last major logistics-related overhauls for "alpha" type status (or even "beta")
Scope: Infrastructure, documentation, project organizing, etc
Current Goal: Keep going, invite in some more help
Phase: In progress
Curator: Cervator
Related: All the logistics things!

Note: The below mega-list is copied out of my Evernote, which includes links (lime green) to other notes. Won't be able to see those here but they're handy for me.


Organizing
  • Go through The Great Convergence thread for more todos (and then mark it as complete)
  • Go through Evernote for more todos
  • Go through Wunderlist for more todos
  • Check on issues still marked as belonging to TGC
  • Review the Nanoware org tracker for more todos (to likely just link from a single issue in the logistics arc milestone)
Applet
Launcher
Versioning / Promotion / Releasing
Gooey
  • Overhaul Gooey - either different bot or enhance hubot - Overhaul Gooey
Jenkins
  • Update all module jobs with streamlined artifact copy (able to use wildcards more) - also grabbing the natives: https://github.com/Nanoware/Terasology/issues/63 - also cleaning properly - https://github.com/Nanoware/Terasology/issues/51
  • Put a bunch of Groovy scripts for Jenkins on to GitHub so we can use them in our Jenkins (eventually hooked to Gooey) - Jenkins-on-Jenkins planning
  • Reorganize the views in Jenkins - maybe ModulesMZ (bundled with game zip, considered "Mode Zero"), ModulesExt (extended modules), ModuleX (special cases, rarely used) - Jenkins-on-Jenkins planning
  • The actual Jenkins jobs that will be used by Gooey (and limited to specific views in Jenkins)
  • Harden job security by overwriting/deleting naughty Gradle files in module builds (checked in versions of build.gradle, settings.gradle - tiny Groovy script in place that can be used to make a sane settings.gradle) - Gradle Redux
  • Update actual job dependencies in Jenkins as a pre-build Groovy step, abort builds that are running out of order - Gradle Redux
  • New facade builder jobs in Jenkins (AWT, applet, PC - last one is already built with engine but needs a separate facade dist job) - maybe https://github.com/Nanoware/Terasology/issues/41relates?
    • Resurrect the TeraEd facade or make it a new DeveloperFacade with an assortment of new utility (should be some more notes here and there). There is also the old chunk monitoring setup
  • Check on forced module errors when dependencies are missing - Check if missing dependencies should cause failures (or warnings / unstable?) even if the dependency is not needed for compilation
  • Options for a highlight build in Jenkins to place details in the in-game UI about what the highlight is all about (if we really need highlight builds anymore)
  • Sort out TODO, FIXME, ASAP type markup tags in code, currently we only use one "priority" level and likewise only one is detected in Jenkins analytics
  • Make a Jenkins build job for TeraMath
  • Figure out why saving a new admin (AbraCadaver) to our Jenkins causes Jenkins fail (outdated? Hit something similar with msteiger, restarting Jenkins made it work)
Gradle
  • Move remaining extra Gradle files into config/gradle
  • Gradle utility task for updating all local modules (loop through all, except Core, do a fetch then a pull) - needs a fix for out of memory situation with many local modules ((or possibly mixed workspaces between ssh/git/https?)
  • Transitive dependencies for modules (likely by making them 1st class dependencies without risking security in Jenkins) - Gradle Redux
  • Support external lib definitions in module.txt
  • Post our project on the official Gradle forum as a complex use case, get advice for how to make it better (maybe we need to write a source module pre-execution Gradle plugin?) - Post our project as a use case on the Gradle forums for guidance?
  • Update the template files for newly created modules - https://github.com/Nanoware/Terasology/issues/53
  • Update existing modules to latest version of everything - https://github.com/Nanoware/Terasology/issues/61
  • See if the cleanIdeaWorkspace task can be made to work instead of a hard deletion of .iws - actually now we need to adopt the new .idea/ approach instead ...
  • Go over the old utility tasks and reintroduce them - such as a super clean that also deletes the saves, config, etc - https://github.com/Nanoware/Terasology/issues/25
  • Speaking of readmes on github - add one to the module template (and make sure it gets along with the module.txt and likely new dependencies Gradle)
  • Move the gradlew refresh target to the utility file? Just to make it clean. Had trouble making that work first time though
  • Put more stuff into the common.gradle under config
  • Go through everything Gradle and reassess the use of << to defer tasks (since it is just a .doLast this may bork config that needs to resolve early)
  • Test updateModule with fetch and pending local changes / conflicts?
  • Find a way to log stats for the update all modules Gradle task (x commits pulled for module y)
  • Make sure updateModule isn't under some circumstances executing for the root project (which in theory is configured, but not enabled, when iterating over the Core module)
  • gradle build file doesn't copy overrides into the build/classes dir before jaring - fixed by Immortius
    Netflix is doing a trick in their Gradle that might allow reusable buildscript blocks - https://github.com/Netflix/curator/blob/master/build.gradle (they also have a neat GH page!)
Site work
  • Module tracking site (where'd @Phillax go?) - Check up on Philaxx and terasologymods.net
  • Possibly use a Terasology/Index repo on GitHub to gather released module metadata + a GitHub Page for that organization to provide a minimal module tracker ?
  • Update Xenforo to 1.2+ Upgrade Xenforo to 1.2
  • Update all the addons in particular the ones we've purchased (separate download) - Update Xenforo Post Ratings 1.6.2 - Update the Post Ratings addon after Xenforo 1.2 upgrade - maybe purchase the anti-spam one, already working great but :appreciation:
  • Introduce use of forum trophies - Forum trophy enhancements including physical merit badge tie-in? - Maybe add in physical "merit badges" (swag)
  • Introduce use of user upgrades (donations-based - a few already qualify, not that we have anything other than badges to give away)
  • Introduce use of thread prefixes (mainly for Incubator, maybe also Suggestions) - initial pass done, need to flesh it out and apply it to existing threads
  • Look into OAuth options for forum
  • Look for other neat forum plugins
  • Consider better support for posting sponsor logos ? Both for tools that grant us open source freebies (YourKit, IntelliJ) and individuals/groups that want to support us (recent interest)
  • Overhaul own forum work thread to go back to where the idea was to share some tasks
  • More screenies and videos for the splash site
  • Update / add some sort of cycling of banners / social media setup to make it easier to update from time to time (unrelated to splash site - FB, G+, Twitter backgrounds and such)
  • Improve backup scripting on webhost - usr/local/backup
  • Improve user setup on webhost (if we get more admins)
  • Find a mirror / alternative setup for the applet? Currently hosted through a URL for the blog on our webhost (might croak from traffic spikes, unlike the released downloads on GitHub) - maybe some sort of elastic cloud offering we can keep at a minimal free level and throttle up if needed? Something like the Cloudflare free level, if that's suitable?
  • Finally make an actual Modding forum ? Review structure between it, the Incubator, and the Suggestions forum - http://forum.movingblocks.net/threads/forum-section-for-showing-off.727/
  • Make a sticky for summarizing "misc" suggestions that do not warrant a whole dedicated thread in the Suggestions forum (and maybe summarize suggestions there overall)
  • Check on the various overview threads
  • Review FreedomSponsors for potential issue tracking tie-ins / sponsoring issues with what little donations we've built up
  • Make more "post rating" icons - Chuck Norris and other crazy ones
  • Can forum trophies be displayed in a user's mini-profile? A.k.a. under the avatar and badges. Some trophies could be randomly handed out for being useful or via trivia questions. And @mkienenb needs a janitor / "rewrites all the things" trophy ;)
  • Consider module categorization and documentation - maybe split Incubator between modules and everything else? Or document modules on a wiki page partially automated by parsing module.txts from everywhere, dragging out type and curator? Link to forum thread prefix tags? Really need a better overview, especially who is working on what (or planning to)
  • Set up a game server on our webhost, now that we can shut it down gracefully from in-game - https://github.com/MovingBlocks/Terasology/pull/1035/files
Documentation
  • Proper JavaDoc publishing somewhere - https://github.com/Nanoware/Terasology/issues/19
  • Enable Javadoc nagging in Checkstyle (including auto-generated stub not being good enough). Able to enable for package-level files too?
  • Bonus: Highlight Modding API related javadoc somehow / auto-generate a ModdingAPI page from said javadoc? Classes with @API ? Module security manager listing? https://github.com/Nanoware/Terasology/issues/36
  • Get some automated docs into the GitHub wiki, maybe again through Javadoc. We have a pile of NUI docs now in the forum, for instance, that would go well in the wiki with some automated snippets included (javadoc links to specific pages?) - Make sure NUI doc makes it into the wiki
  • Modding guide improvements in the wiki - especially BTs, blocks, inventory, multiplayer, annotations, and so on - Modding Guide: Behavior Trees - Modding Guide: Blocks - lots of doc scattered all over the forum, of course
  • Update DevSetup and AlternativeSetups in GitHub wiki - Update DevSetup and related pages + especially AlternativeSetups - maybe include a note on optionally using GreenShot with imgur for screenies. Also maybe a common issue for a full IntelliJ restart needed in some cases - Doc new Common Issue - IntelliJ full restart needed in some cases
  • Automate wiki doc for file extension types ? Wiki doc more file extensions
  • Better changelog and "known issues" by release - Launcher did some structure there - Review the launcher's change log and see what we can do on the engine side - also had an old idea about auto-updating a wiki page with commit messages but then allowing human edits for a more readable version
  • Update GitHub wiki with new "GitHub for Windows" type setup and https:// instead of git:// Update GitHub wiki pages with todos and such in place
  • Flesh out CodeBase Structure if anything remains (left some todos there at one point) Update GitHub wiki pages with todos and such in place
  • Game asset auto-doc for wiki, also a way to better catch text vs binary for git (involving .gitattributes) - Add to asset type doc / update .gitattributes
  • Update the Contributor Intro page in the wiki
  • Update the Contributor Intro forum - needs a primer on teams and badges + benefits of contributing (access to our resources like IntelliJ Ultimate, YourKit, etc)
  • Add http://forum.movingblocks.net/threads/blender-to-md5-howto.1020/ to GitHub wiki and link to from all over - was there a GitHub issue for that too ? Also mention the similar COLLADA stuff
  • PR guidelines / CONTRIBUTING doc somewhere - Make an outright page in the GitHub wiki for PR guidelines (think the Launcher again has something)
  • Update doc about our issue tracking (one node in the forum, probably a page in the GitHub wiki also?) - don't forget that you can view all issues for a whole organization on the right page on GitHub, useful. More details in Wunderlist (focus)
  • Better (minimal) doc for GitHub for artists and others that may not need a complete Git setup (using the site purely through the site?) - has an issue under L&S
  • Automate doc about where the game data directory goes, depending on OS? Or just display it better in-game (if we automatically pull it from code and update howtos in the wiki that might be ideal from an "up to date" perspective). https://github.com/Nanoware/Terasology/issues/30
  • Write some doc about Jenkins itself and how to set up jobs properly (hit-by-bus insurance)
  • Write similar doc about Gradle-fu, in particular how defining tasks differently impacts whether or not they auto-execute whether they're called or not
  • Actually doc some reasonable system requirements for Terasology somewhere (in particular maybe identify the minimum OpenGL version?)
  • Add our custom json extensions to settings / file types for MOAR HIGHLIGHT in IntelliJ (thanks for hint @MarcinSc !) - turns out the javascript plugin in Ultimate is what makes it available
Misc
  • Normalize LF / CRLF in all text files in the distributed app zip and modules - https://github.com/Nanoware/Terasology/issues/49
  • Can we include our own entries into FindBugs or catch them otherwise? Example being entity leaks like that inventory scenario - Tweak at InventoryUtils ? And avoid that entity leak
  • Overhaul project testing strategy - split into proper separation between unit testing, integration testing, benchmark testing, etc - probably including supporting plugins in Jenkins - Figure out the next phase of testing structure
  • Code coverage setup ?
  • Fix the PMD issue where it isn't ignoring protobuf properly
  • Maybe find a way to offer shirts, mouse pads, or something? SuperSnark made us some shirt designs already, and i think somebody did mousepads/mugs or something
  • Final check to make sure the LogBack Groovy thing is in place everywhere it is needed, including for testing (test/resources for all modules? included in template?)
  • Better scripting / addons for model conversion needs (maybe tie into a bigger overhaul with a newer format period?) - Make sure we have full doc on the MD5 issue so it can be put to bed
  • Improve TeraMath in Jenkins/Artifactory - TeraMath updates - like basic release promotion non-snapshots
  • probably do a video on Terasology's console (command cycling) as well as the new chat console, player settings, and multiplayer
Lower priority / WIBNIFs
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Oh am I ever re-remembering how I both hate and love Gradle ;)

After beating my head against a wall getting used to the quirks for several hours tonight I finally returned victorious - with an understanding of merely a couple tiny but important things and with a single small new utility task

Code: 
gradlew updateModule
Behold the glorious complexity of this massive command! All four or so functional lines of it.

And be sure to run "gradlew refresh" once first in existing workspaces as this change is in the build.gradle for modules, which is copied out from Core when you fetch new modules so those copies won't update on their own - the refresh handles that.

In short it does the default "git pull" for all active modules, excluding Core. Doesn't update the root project, facades, or anything else. Just every active module you've got locally.

If an update brings in a new dependency (via module.txt) that still requires a second execution of Gradle to recognize next. Baby steps and all that.

I haven't tested it against unfetched repos (git pull is supposed to also fetch, but I've grown superstitious) nor with local changes in module repos that aren't on GitHub yet. It might auto-merge or complain about conflicts. Best used in a "clean" workspace after you've pulled engine.
 
Immortius

Immortius

Lead Software Architect
Contributor
Architecture
GUI
I should just copy my NUI documentation to the wiki. I just like to use the forum as a staging ground.
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Yep, agreed - forum is good for discussion and forming documentation :)

Still more stuff to do like connect everything to Javadoc better. I am tempted to be evil and configure Checkstyle to demand Javadoc everywhere that's sensible.
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
I pulled what use I could find out of my temp applet working directory from some while ago

https://github.com/MovingBlocks/FacadeApplet/commits/master

I had pulled out an old build.gradle for the root project before The Great Convergence when we had the applet's index.html somewhat generated with information on project libs, modules, and so on. While just updating things I had been working with pre-compiled artifacts (the Terasology.jar here contains the applet class) in a manual directory, placing all libs (including engine) in /libs, all modules I cared about (just a few samples) in a mods directory, and hacking the index.html back to a hard coded list of libs and modules - that got me to where I could at least attempt to launch the applet, then hit the new security features. That's where I stopped since I figured we'd have to come up with a legitimate certificate.

Manually_2014-02-18_00-57-45.png


I put the old build.gradle into the FacadeApplet in one commit for reference, then hacked off all the extras in another - just enough so that you can fetch and set up the facade.

Code: 
gradlew fetchFacadeApplet
gradlew idea
Or Eclipse etc.

This is mainly in case msteiger wanted to play with it real quick sometime, getting the updated applet manifest thing put together and so on. The default keystore in there has our old self-signed certificate, we'll replace that shortly with a new individually signed cert :)

I'll get back to it myself otherwise when time presents itself, soonish(tm)

The tricky part is compiling this solo, rather than in a multi-project workspace where you can depend directly on the engine and such. Like the Core module's build.gradle we need to handle it two ways, one way with direct dependencies and one way from Jenkins receiving the engine harness. In that case we probably extract the zip, then build the lib jar list from /libs and the module list likewise

But I figure we can deal with solo builds second, after getting it working again first. The only compile error right now is from the whole subsystem overhaul, which came after I had set this up the first time. Most the work will be Gradle-fu
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
.. uh, that was sort of anti-climatic. Was getting re-acquainted with the outstanding Gradle issues, like not being able to load the project in IntelliJ via Gradle. Noticed a stray error with just a tad bit more info than I remember from the past, related to Gradle getting lost in the JDK home instead of using the project path. Got the crazy idea to qualify the path in subprojects.gradle a little better, including the rootdir now:

Code: 
new File(rootDir, 'facades').eachDir { possibleSubprojectDir ->
... and then it worked. Loaded perfectly in IntelliJ in full Gradle mode. So it isn't trouble with the overly dynamic structure - it was a quirky "IDE wants to make this path relative to this other directory"

Well, perfectly is a bit much. The project structure loads and looks correctly, however the IntelliJ customizations (the whole idea {} block) seems to be completely ignored - maybe even explicitly as I can see the "eclipse" task in the JetGradle window and its relatives (cleanEclipse, etc) yet all the "idea" tasks are completely missing (but everything else is there).

However, functionally all you need to do is swap Project Language Level from 1.6 to 1.7 (at least on my system it defaulted to 1.6) and run extractNatives (which is normally triggered by the idea task). Then the game runs normally - although without memory settings, -homedir, etc. Will be looking at that some.

With that surprisingly out of the way (I wasn't even trying to fix it!) the dependency fun is the other big thing for now, especially source module dependencies / fetching stuff without breakage. With a new strategy in doing it moar Gradly I think most won't be too bad, although I'm sure I'll get stuck on a series of small issues that take hours to sort out.

Will also try to sort out the overrides in binary form - thanks for the pointer Immortius !

PrivateAlpha - ping! Since we were looking at this once and thought we had to hack the IDE plugins or something before getting it working :D

In other news we have an official code signing certificate now. Just need to fix up the applet and apply it. Will give it a little time in case somebody else wants to play with it a little, nudge nudge wink wink you know you know msteiger :trollestia:
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Had a decent time with Gradle, Jenkins, and Artifactory over the weekend.

I've set up a complete testing framework in parallel from our usual jobs: http://jenkins.movingblocks.net/view/Nanoware with matching forks on GitHub

That includes the whole WoodAndStone / Throughout The Ages line-up from Marcin Sciesinski since that's the fanciest dependency situation we've got

Everything publishes to org.nanoware in Artifactory and resolves from there as well, should be entirely independent from org.terasology and the "live" modules

Now I'm beginning to play with a new "dependencies.gradle" in modules, publishing them to Artifactory with dependency info (well, when that works), then in theory you should be able to just fetch, say, CopperAndBronze, and Gradle will resolve all the dependencies including transitive ones (CaB depends on WaS, depends on PlantPack, depends on GrowingFlora ...)

That duplicates info available in module.txt, so I'm thinking about maybe having the game parse out module dependencies from Gradle rather than having Gradle parse out from our arbitrary game format. Still, I may have underestimated how well Gradle was able to handle said parsing, with some stuff just not being enabled yet. Lots of stuff to still figure out and get the best design for. There is some release management hiding in there too. Adding third party libs to modules would also be easy, not that we couldn't also add that to module.txt

I also found https://bitbucket.org/nm2501/holy-gradle-plugins/wiki/IntrepidPlugin which as it sounds may be the holy grail of Gradle in our particular context ... as it allows you to define remote "source" dependencies, letting them fetch from an online repo, then process locally, before moving on with Gradle's execution flow - which may mean single-task module fetches + project structure refresh without having to run Gradle a second time. That's somewhat lower priority though and a fair bit more complicated.

For now I'll focus on cleaning up dependency management, which will include having things in Jenkins build/rebuild in the right order (engine job causes all modules to build, any module with a dependency on another module will rebuild (or only build) after that other module has built, etc ...)

While I was playing around I set up a new job for TeraMath, so that's getting built and published to Artifactory too :)
 
Mike Kienenberger

Mike Kienenberger

Active Member
Contributor
Architecture
GUI
Cervator said:
That duplicates info available in module.txt, so I'm thinking about maybe having the game parse out module dependencies from Gradle rather than having Gradle parse out from our arbitrary game format. Still, I may have underestimated how well Gradle was able to handle said parsing, with some stuff just not being enabled yet. Lots of stuff to still figure out and get the best design for. There is some release management hiding in there too. Adding third party libs to modules would also be easy, not that we couldn't also add that to module.txt
Click to expand...
I don't think we should have our module dependencies defined in Gradle. We shouldn't be tying our application to our development environment. If all hope seems lost, let me know, and I'll spend the time to figure out how to make a gradle plugin to do what we need.
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Mike Kienenberger agreed and I already realized that since my last post (actually before falling asleep that night, heh). I was still stuck on thinking I had done stuff with more quirks than Gradle could handle, but it is totally fine with us using our own arbitrary format so instead imma put more fun dependency stuff into module.txt! :D

I got transitive dependencies working with Artifactory, Jenkins, and local Gradle, although with mixed success as JFrog (Artifactory maintainers) ironically seem unable to get their own versioning right, with the primary config example referring to a newer version of their Gradle plugin than I can find anywhere on the web ;)

If I try to use the Jenkins plugin for Artifactory at the same time as having anything Artifactory configured in the Gradle script they version clash and so on. I might just have to try doing everything in straight Gradle using whatever stable version of that Artifactory plugin I can use, or even the Maven publishing setup that seems kinda-sorta able to do most if not all the work.

I'm very confident now on getting everything working in a relatively elegant fashion, including still no Gradle in the module repos and thus no need to go crazy doing some sort of sanitization process in Jenkins :)

Got some more notes and updates for the original post here but I'm going to see if I can get something working for GitHub soon'ish(tm)
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Round one is pretty stable at this point and likely merge-worthy soon with some review + polish. I've tightened up module dependencies and done some prepwork for later applying version/release promotion and split jobs in Jenkins between develop/master (going theory: unstable + released for modules)

First big change was moving from modules blindly depending on whatever they got in Jenkins as part of the "engine harness" (artifacts attached to and copied from the engine build job into module build jobs) to full Artifactory usage even for the engine and core jars (and engine-tests, soon PC jar, other distribution fun)

Here are some builds jobs showing logging + check out Artifactory and the poms there:

http://jenkins.movingblocks.net/view/Nanoware/job/NanoStable/19/console
http://jenkins.movingblocks.net/view/Nanoware/job/NanoFences/4/console
http://jenkins.movingblocks.net/view/Nanoware/job/NanoAnotherWorld/17/console
http://jenkins.movingblocks.net/view/Nanoware/job/NanoGrowingFlora/5/console
http://artifactory.movingblocks.net/artifactory/webapp/browserepo.html?6&pathId=terasology-snapshot-local:eek:rg/nanoware

And here's a look at local logging with all the W&S modules I cloned (a few have been added since but I'm not updating the forks)

...
Processing module 'Hunger' in a multi-project workspace
* No extra dependencies

Processing module 'Journal' in a multi-project workspace
* Journal has extra dependencies:
** AnotherWorld
*** Found E:\Dev\Terasology\Git\Nanoware\Terasology\modules\AnotherWorld locally

Processing module 'MultiBlock' in a multi-project workspace
* MultiBlock has extra dependencies:
** Core
*** Found E:\Dev\Terasology\Git\Nanoware\Terasology\modules\Core locally

Processing module 'PlantPack' in a multi-project workspace
* PlantPack has extra dependencies:
** GrowingFlora
** Crops
*** Found E:\Dev\Terasology\Git\Nanoware\Terasology\modules\GrowingFlora locally
*** Found E:\Dev\Terasology\Git\Nanoware\Terasology\modules\Crops locally

Processing module 'WoodAndStone' in a multi-project workspace
* WoodAndStone has extra dependencies:
** PlantPack
** MultiBlock
** Journal
** Fences
** Workstation
** Hunger
...
Click to expand...
Yep, WoodAndStone "only" has 6 dependencies in this setup, rest are transitive :geek:

Biggest changes are probably in the Core module's build.gradle since that gets used for all the modules. There is also new stuff in the config dir for Gradle, mainly the artifactory Gradle script

I'm waffling a bit on whether to merge this sooner or first straighten out cascading builds in Jenkins, with a final distribution job that grabs latest modules after they've finished doing the builds the engine build itself triggers. I've got a nagging feeling that the tighter coupling of exact versions may cause trouble somewhere - but probably not until module.txt versioned dependencies are respected (so far, and for round one, the latest version is always fetched regardless of version requested in module.txt)

Lots more to do, but this is a good start. More technical details available on demand / on IRC, but it is late and I am sleepy :)
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Okay, went through and cleaned up the from-source version some more now. It wasn't handling a mixture of source + binary well when transitive relationships were active

Right now I can fetchModuleWoodAndStone with a few of its dependencies present in source form and it'll appropriate pull everything needed from Artifactory and leave runtime copies of the jars from the Gradle cache to under /modules - there are some edge cases though:
  • If a module build runs in Jenkins that you already have a runtime binary for (as in, you don't have the module in source form locally) you may get the new binary as well. Game just ignores the dupe at runtime but I'm not sure which it ignores. Probably out of scope for the current work.
  • If you have a source module A that has pulled a binary dependency B yet that module has a dependency C for which you have source I'm pretty sure A will ultimately build against a binary version of C fetched to the Gradle cache, yet at runtime you'll be using the source version of C
  • If you then delete the source for module C Gradle will correctly catch it and prepare a runtime jar instead
  • Gradle/IntelliJ is resolving "-sources.jar" artifacts wonderfully and will show you source classes from a module you only have in binary form if you look
  • However it will also retrieve the engine source jar (and Core, and other things) even if you have them in actual source ...
So still a bit more testing needed but it is 3 am and tomorrow is a work day :D

Marcin Sciesinski - want to give it a spin with Wood & Stone modules in some odd combination? Immortius would you like to do a sanity check of local behavior & logging being usable? mkalb - I don't think I've broken anything for the Launcher yet but I probably will eventually. At this point the engine job is lacking plain jars (engine, libs, etc), but it still has the unchanged zip + the extra version file: http://jenkins.movingblocks.net/view/Nanoware/job/NanoStable/lastSuccessfulBuild/artifact/build/

To test from source:

git clone https://github.com/Nanoware/Terasology.git
gradlew fetchModuleWoodAndStone
gradlew idea
Click to expand...
Then feel free to do more fetchModules and delete existing source dirs / fetched runtime binaries + rerun plain "gradlew"

Everything in the Nanoware forks is isolated from MovingBlocks and will be using slightly older W&S modules: https://github.com/Nanoware
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Minor update: Added the engine-tests and PC facade to Artifactory to get them from there when needed rather than attached in Jenkins. I'm going to see if I can sort out a better assembling of the PC game distribution, and at the same time take in the applet updates from msteiger as well as get Jenkins going for the AWT facade Mike Kienenberger put together. I've forked both of those to the Nanoware org and will work with them in Jenkins next.
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Round one has been merged

Transitive dependencies should now work in a local workspace, as in Jenkins - but I might have gone out of sync somewhere. The engine build works fine, AnotherWorld built fine, GrowingFlora failed - I still need to do some cleanup in Jenkins and Artifactory, it may be due to that (and the Nanoware engine build is still behind upstream - merged forward via a rebased integration branch). Too tired to fix tonight, 1 am + work tomorrow and all that

I did a stable build right before merging, so the master branch is immediately before the Gradle updates

Marcin Sciesinski - feel free to start removing transitively resolvable dependencies from your modules (and in some places add in dependencies previous covered by the catch-all in WoodAndStone). I'll catch most or all of them while updating Jenkins, but I may need a note or two from you since you added four modules just while I was branched off ;)

In better news said 4 modules all resolved beautifully when I hadn't even thought about needing them, heh.

Also, have a random screenie:

Terasology-140413141502-1152x700.png
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
So that was weird. Initial problem was easy, modules weren't resolving the very last snapshot version so stuff broke due to using pre-change dependencies.

But then the engine build started failing during the javadoc step for the PC facade - seemingly due to scanning the just-built engine-source jar for inclusion in Javadoc (maybe related to this link - same error message at least). Yet a complete clone of that job in Jenkins ... worked fine.

Tried an assortment of cleanup in Jenkins, even restarting it, no change. Maybe it somehow relates to the plain "Terasology" job's high build number. Or something in Artifactory still being messy, but that seems unlikely since that job just builds stuff, doesn't draw on Artifactory for Terasology stuff.

Either way the PC facade has all of exactly one class, so separate Javadoc for that might be overkill anyway - disabled it for now. If I can reproduce it elsewhere maybe I can poke at it some more during facade work, but for now Jenkins is angry enough as-is :)
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Finally categorized the ridiculous todo dump in the first thread. Yay! Also did some cross-referencing and poked at issues on GitHub a bunch.

Will probably do another pass along with the Alpha thread soon'ish. Been pouring issues into the Alpha milestone on GitHub too.
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
All "stable" module builds in Jenkins have now been updated to the latest design and rerun until all is working. Fixed a single engine-tests issue breaking Pathfinding, and still most its unit tests but that's on me and natives detection.

The latest dev build contains all the modules and they look to be working well. Time for a stable build soon'ish methinkss :)

Marcin Sciesinski - could you do a review of your dependencies to see if they're all well, now with full transitive support? For instance:
  • Should Genome still depend on Core, or is the dependency on AnotherWorld (which depends on Core) enough ?
  • Does PlantPack need both GrowingFlora and AnotherWorld directly?
  • WoodAndStone still depends on all the things - too many or just right? CopperAndBronze?
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Unstable modules also have had their jobs updated now, although naturally some of them won't build successfully. Some are still outstanding from the module overhaul and likely easy to fix, others a tad more work.
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Update - although one without tweaking the top mega-post with completed items, too late at night for that right now :)

I'm tinkering with Jenkins and have successfully added job (build) dependencies programmatically, opening the way for module builds dynamically updating their job dependencies in Jenkins based on module dependencies in module.txt. I also think I may be bleeding Jenkins code out of my ears after this weekend.

Goal is to run that on every module build caused by a Git commit (or by manual execution, or some other cases I haven't narrowed down yet) so when the author of module X adds module Y as a dependency Jenkins goes and updates the parent job (Y) to build the author's module (X) after each time it gets built.

In other words this process updates the dependency tree in reverse, so next time Y builds it naturally triggers a build of X to see if it still builds successfully (and if not the author gets notified that a dependency has changed and fixes might be needed)

Long story short - this will allow Jenkins to build all modules ever in correct order after each time the engine is built. There are ways to prevent downstream jobs (even transitive ones) from not building if an upstream job is in the queue (X depends on Y depends on Z - only Z will be allowed to build first, then Y, then finally X). So all the Wood & Stone modules for instance will build in the right order each time.

On top of that I've also got template jobs working to where there's finally only one spot to modify for all module builds ever (at least the build steps and post-build steps - not yet the Git setup, Artifactory, etc). That'll easy any future tweaks to make all this work everywhere.

As part of this big pile of stuff I also hope to do some related changes like finally getting third party lib jars supported in modules so poor Skaldarnar can finally play with Scala and Antlr properly :D

For now, sleep!

Edit: Oh, bonus. Due to awesome timing the way Jenkins inter-job dependencies work is changing just a few versions from our current level of Jenkins (1.555, big changes in 1.560 and several others involved too). It is new enough that doing it the "proper" way is still buggy, yet using the existing way is more hacky, and will likely need to be rewritten later. So all kinds of confusing signals getting crossed in the related research and javadoc surfing. Fun! Luckily it will only be one or two scripts, so not terrible to tweak later. Next Jenkins version update will need extra testing and care, it may be wise to stay at 1.555 a while.
 
Cervator

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Have begun another round of big changes.

I'm testing locally from source, via a local Jenkins, and via the Nanoware jobs in the main Jenkins. Not done yet but first batch is looking good and I've got two more days of long weekend to go :)

I've finally gone through and fully understood how the Artifactory release plugin in Jenkins work, same with the plain release plugin. Also been playing with Artifactory config some more. I think we should go with something like the following:

Artifactory release management works for gestalt-module where the plugin updates a gradle.properties that is the true source for versioning info. This includes staging/promoting a release which at first publishes to libs-snapshot-local in Artifactory, then promotes to libs-release-local. Although I'm still a little puzzled between the Artifactory Pro requiring method vs. the basic one that still seems to have correctly promoted from snapshot to release.

This is the ideal approach to also use for other libs, that do not get into our module.txt fun. And I think all those libs should go into the libs local repo in Artifactory.

Normal Jenkins release plugin should work for the more centric Terasology pieces with module.txt in place instead of gradle.properties. This is in part because of the extra hassle it would take to make the Artifactory process work. You'd first update a gradle.properties with version info, then hack a matching update to the module.txt. Second part is that we'd need gradle.properties files for module repos, which IMHO is a possible attack vector on Jenkins - a naughty module could introduce properties and possible override things in the build system. And we hand out module access like candy specifically because we can sanitize the repos safely in Jenkins.

With the normal release plugin we just get some extra steps to do on a build we sanctify as a release, which could trigger a build on a master branch + job configured differently. This is the approach I had thought about earlier, although I was very close to jumping on some sort of one-job solution instead, the Artifactory plugin (and a similar plugin alternative) are very close to sufficiently customizable, but not quite.

A master job could have different settings for publishing repo and resolution repo, forcing only the use of fully released stuff, no snapshots, and sending products to a release repo instead of a snapshot repo. For example I built a Nanoware fork of MultiBlock aimed at a new nanoware-release-local repo in Artifactory, while a main engine build is going to nanoware-snapshot-local. Still working on how to make the resolution repo setting in Jenkins override the default defined in common.gradle, which is currently simply "repo" (same for existing Jenkins job), which is a virtual repository mapping every other repo in Artifactory

We can make other virtual repositories instead and use them accordingly. For instance I'll probably set it up so the Nanoware repos are excluded normally. We can have one like that which includes (in order): external library repos, internal library repos (libs-release-local), Terasology component repos (terasology-release-local)

I think we should stick with separate terasology-release-local + terasology-snapshot-local just for the organizing (distinction between libs using full Artifactory and Terasology stuff using homebrew) and I think resolution could be made much better. Right now the sole engine "release" that @msteiger put into Artifactory so the WorldViewer can get a stable artifact for its uses is actually also getting pulled by snapshot module builds because of that global "repo" thing plus release > snapshot. Might also depend on some version number quirk, dunno yet.

I de-blacklisted the terasology-snapshot-local and temporarily let it work for releases too, until we can get around to build released versions again and/or more properly allow local development to use snapshots. Not entirely sure what we'd want there, but something configurable for sure. Jenkins should be able to override like it currently can for publishing to a specific repo. So things should be back to normal while testing and ready to break things again / cleanup the snapshot repo.

On the short list right now:
  • Look at overriding the resolution repo in Artifactory from Jenkins to better control that
  • Look at excluding some dependency definitions so the engine doesn't resolve when it isn't needed (like locally when you have engine in source form). Might copy in some artifacts to module jobs to use instead of Artifactory so unit tests don't try to start out of the Gradle cache.
  • Look at making master jobs that will build exclusively from release repos in Artifactory, ignoring all snapshots, and publish releases. Unless there's still a single-job solution. Might impact change listing in the Launcher, even release builds on a release job would end up with executions containing nothing but a version bump? Unless the master job is what you run to draw in changes from dev ... hmm
  • Look at doing the bit of Git-committing the Artifactory release plugin does in our own script to correctly handle increasing version numbers
  • Look at applying a Git tag at the same time. Still tempted to test Git commit since last tag as dynamic patch number, but that might be too convention-violating :)
  • Tie it all together with the Release plugin running extra steps
  • Hook the "Distro" thing in to then also package the game with modules in various formats. Already tested.
@msteiger - how far did you get with Git writes when you worked on the Index piece? I forgot if you got to where you were committing stuff back to Git, or only reading

@Immortius - does this all sound like an OK approach to pursue or did you have stronger reasons to prefer a single Artifactory repo for releases and one for snapshots? We can effectively have that still with virtual repos

2 am disclaimer: It is past 2 am. I'm sleepy. Brain fog may have clouded my ideas above, feel free to correct / provide feedback :)
 
msteiger

msteiger

Active Member
Contributor
World
Architecture
Logistics
A few assorted notes:
  • Separating lib builds from TS modules sounds reasonable
  • Separating snapshot from release repos. sounds very reasonable :cool:
  • For modules, I'd like to pull out the version number from module.txt if possible (maybe add a snapshot extension - can we add classifiers to module.txt versions?)
  • Not sure why the gradle.props hack is necessary. Might need to read it again :)
  • I believe that release mgmt. / promotions are best done through the gradle cmd-line (artifactory-publish plugin?) , not Jenkins. This can be done easily together with creating git tags.
  • About the PR: I have the gradle settings in my home folder. Why do you add the template to the code base? Do you want separate configs for the projects?
  • The "ModuleIndex" repo contains the JGit code. It also supports pushing updates to github. Personally, I prefer the manual approach - I always clutter the repo with auto-commits :rolleyes:
  • Maybe the whole thing is easier to sort out on IRC (or Skype?)
 
You must log in or register to reply here.
Top