Design Module Tracker (infrastructure)

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Rough draft Second draft design for the infrastructure-centric side of handling modules of varying versions.

msteiger is already interested in playing with it some. Feedback appreciated! As usual this is a proposal up for debate, not a design written in stone.

Summary

Use a single GitHub repo as an "Index" for available modules then have a library project to query said repo as a sort of repository database (like yum) to display available modules to a user in various scenarios (Site, Launcher, Game).

Index repo is expendable - it mirrors information available in Jenkins (released module jars) and in the GitHub module repos, but is a single central spot to query rather than iterate over all module repos under a specific GitHub organization (or worse: hitting all module jobs in our Jenkins or jars in Artifactory every time).

Offers a framework extensible by third parties who can create their own index type repos and have users hook up to those in addition to / instead of the official repo via configuration in launcher or in-game. Probably also reusable independently along with gestalt-module (gestalt-index?)

The Index

This is a single special repository under the GitHub Terasology organization (where all official modules live). In it each released module gets a module manifest similar to the module metadata file (module.txt/info), but with added per-release information like version number, jar file download link, dependencies as of that version, and so on. Similar to a GitHub-based version of our Artifactory, really, but backed by GitHub's infrastructure.

I figure a one JSON file per module approach should work. It'll only be updated when a release is promoted in Jenkins, not for every snapshot build, so updates shouldn't be super frequent (at least not without a massive amount of modules). Alternatively maybe each module gets its own directory with a single file for global stats + one file per release. Not sure what is preferred / more efficient.

The Index itself should be portable so it could live and be queried somewhere other than GitHub. It should define a format others can replicate to create competing index repos. Users could then add those in as alternative sources for modules (with an appropriate disclaimer about using third party modules that haven't gone through official channels)

The Site

An extremely basic module tracker site will live as a GitHub Page, no rating, commenting, or anything fancy available there, just manual browsing/downloads offered. This leaves space for a more feature-heavy custom site hosted elsewhere, yet even in the case of a traffic spike overwhelming the fancy site the basic site (and Index functionality) will remain available.

I am looking to the Netflix OSS index page for design inspiration here as it does per-repo stats and looks awesome. Imagine that, but for modules. I haven't dug into their setup enough to know if they are pulling repo stats live to build the page or if they also "cache" the stats somewhere first. While pulling stats like for 10-20 repos live would probably be doable I doubt it is for hundreds or even thousands of repos.

There should be a way to hide, dim out, or put disclaimers on modules that go inactive or break vs latest engine version. Or maybe highlighting version info would be enough, can then gradually fade entries if they haven't had new releases as new engine releases come out. The Index repo probably can know a little extra about the engine (or rather, the jobs in Jenkins that do the updates will know). Like the Index it should be possible to generate and host the site off GitHub if needed.

Probably the site should live in a second special repo under the Terasology org since it needs to be named something special, that way the Index repo also remains as small as possible. Yet it really needs all the info in the Index to be able to build the site, so I am not sure. Maybe it just contains the logic and then loads data from the Index repo.

The Tracker

A separate library project should be created under MovingBlocks on GitHub that can pull down and scan the Index repo. Downloading the whole thing as plain text (or as zip, really) shouldn't be awful, the repo should never have graphics or anything intense, at least not per-module. The tracker should then parse out all the release details in some sort of module data structure that can be interrogated by the calling software. Examples for this include:
  • Official launcher. Add a new "modules" tab of some sort and allow pre-loading a set of them. This'll become more useful with more advanced launcher features, like automatically setting up a headless server including modules + config (I bet you could even set up a remote server via SSH!). There could also be a separate "News" section specific to module releases.
  • Terasology itself. Likewise allow a list of available modules to be shown somewhere and be downloaded before creating a world. Could perhaps also be used server-side when it needs module updates before sending them to users after a version upgrade
  • Fancier module tracking site. Akin to what Philaxx was working on - a full-fledged module site with reviews, comments, ratings, etc
Only when a user requests install of a set of modules will the actual module repos be hit with requests for downloads. Until that point everything is purely based off the Index data.

Workflow

The Index will be maintained by Jenkins and depends on the full release management implementation for module build jobs. Ultimately that will result in a promotion approach to stable releases when a module author is ready to release. When a release build runs it'll trigger a series of extra steps not done for snapshots - some/all may be via Groovy script(s) rather than Gradle as it isn't exactly super flexible for when you want to do crazy things:
  • Author triggers a promotion for module X via the normal "Unstable" build job (tied to develop branch)
  • During module release promotion any needed version & doc updates are done and committed (develop -> master)
  • "Stable" module job (tied to master) triggers on-commit with final release info
  • Jar from master is published to Artifactory's normal release repo rather than the snapshot repo - only developers and other jobs will actually get files from here
  • Jar will also upload to GitHub's "Releases" tab (there's an API for that) where users and module tracking will get files from
  • Stable job may do some extras like updating a GitHub wiki page with doc updates, updating a Xenforo thread header with the latest module info, notifying for any "New release available!" needs, and so on
  • On the stable job finishing the release from the master branch it'll trigger an Index update job as a post-build action, passing needed parameters. It'll also commit back to the develop branch (version update for next unstable + changes made in master) to trigger a new snapshot build
  • Index update job is Git-configured to the Index repo rather than individual module repos. Will modify the appropriate module's manifest in the Index, then commit and git-push an update
    • Unsure if this would also need to trigger an update for the site repo, if it isn't the same repo or loads fresh data from the Index repo each view
Gooey will have options to do some of these tasks via IRC, but will really just be fronting jobs in Jenkins (primarily trigger the promotion step - initially creating a new module repo doesn't yet do anything to the Index repo, only releasing a module does)

Most likely there will only be a single job in Jenkins responsible for updating the Index, pointing to the Index repo via Git, and triggered by the module jobs responsible for doing releases. It won't run on-commit, only indirectly. This gives us a nice spot to put related scripts and prevents multiple jobs from trying to update the Index at the same time (instead they'll queue up separate executions of the Index updater job). Parameters can be passed from the module release job, could even include module.txt or its contents (after release edits are applied)

Another related activity in Jenkins is post-engine release module tests. If an engine release runs it would be good to know if all the currently released modules work at that version, as well as the very latest module snapshot.
  • On engine snapshot build: Trigger all module snapshot jobs to see if they still compile so authors will know early if upcoming engine changes break their stuff (already a goal)
  • On engine release build: Test or do something with the released modules to see if they still work at their latest release or even a few versions back with the newly released version of the engine (new idea). Looking to somehow create a compatibility matrix (or other "decoration") in the Index repo, but this is probably a luxury item for later (beta level?), like when we can run some sort of integration tests separately against existing artifacts.
Distros

One of the "meta" things we can do in an Index repo is define a few recipes of engine + some set of modules == some flavored distribution of the game. Currently we just have a manually maintained list in Jenkins' engine build job, which needs to be split off to a later phase and made less hard coded.

I figure this can be done by maintaining a few lists in the Index repo, possibly with functionality advanced enough to exclude a module or a module set if build errors occur on release (might mainly be a problem after engine releases causes module rebuilds). It could also be used to offer a lighter bundle for the applet, or a distribution of the AWT mode with just a few relevant modules. Still wondering if this is a good idea or not. Potential distros:
  • Bare game: just engine + facade, no modules, can't create a world (none available) - mainly for the Launcher to use as a base game download to then stack modules on top via user choice
  • Mode Zero: engine + facade + MZ modules - the set of modules we consider part of the "core" game platform (inventory, basic world, basic player, basic creature AI, etc) - can create a simple world with basic gameplay that is the primary foundation to build more advanced modules on top of
  • Applet: engine + facade + minimal modules to showcase the game. Would likely lack most the soundtrack to keep the size small, but highlight a few of the more interesting bits of content like Wood & Stone, Light & Shadow, Cities, etc
  • AWT: engine + facade + few AWT-friendly modules. Terasology-lite / DF style with less intensive need for video card performance
  • Fat client: engine + facade + ALL stable modules. Essentially the current game download zip.
Each of these would be tied to a distribution job in Jenkins that would run after engine builds (and all triggered jobs thereof) finish. As such those jobs might be able to run off the Index repo directly, but only trigger indirectly (and in parallel) from other jobs in Jenkins (Index repo would have no on-commit builds for itself). Possibly we could include options in Gooey to "flag" modules in the Index repo on request to be part of some distro, could also determine tabs/views in Jenkins

This I think would be a huge help in defining distribution jobs in Jenkins, which has caused me some design headaches in the past.

Backwards Compatibility

Another utility option in the Index repo would be to change URLs, module names, etc in the metadata file there to maintain compatibility over time (since you could make that change as-of a particular release)

Say a module gets renamed from X to Y, on GitHub the module repo just gets renamed as does any affected files in it. Unless we leave behind a "shell" repo with a forward any old references would break. If a new module is ever released with the name of a different old module - all hell could break loose.

This (or some of it) may be better tracked in the actual module metadata file (module.txt/info) but in the Index this could include changed download locations and such as well, per release.

This is a lower priority piece of the greater picture. I'm also not sure how you would trigger a "rename" - if there even would be anything to trigger, maybe just updating the Index with the next release looking different would be enough combined with smart module scanning.

I also wonder if this might be helpful in maintaining backwards compatibility through world changes, but that's pretty much just speculation at this point

Related:
  • Module categories - for decoration / display
  • Release management - need to only run stuff on releases, not snapshots
  • Expansion of Gooey automation - mainly to allow promotion (and thus index update) on bot-request
  • Gestalt-module - since we might be able to leave the setup extensible for use outside of Terasology (and maybe the Index work or Tracker need that level of knowledge about the module setup)
 

msteiger

Active Member
Contributor
World
Architecture
Logistics
I wrote a gradle script that updates a github repo based on a local module.txt file. The remote repo is this one:

https://github.com/Terasology/Index

This is what it does:

It tries to open a local git repo. in 'module-index' and pull latest changes. If that fails it clones from github.

Then, the "module.txt" file is copied to that repo. and the local README.md is re-created (with a nice table, woo!).

As a final step, changes are staged and pushed to github.

Questions/Issues:
  • Where do we keep this script? In Terasology / config / gradle ? A new repo?
  • GitHub is probably not too happy if we upload lots of binary jars. Why not keep them in Artifactory?
Next steps:
  • Setup Jenkins to run the script for all modules
  • Add download size and URL to module.txt before uploading - we need to know the download URL before uploading though :(
  • Convert README.md file to github page
  • To use that info in-game, we probably need to setup a webserver at some point of time (which could also list available games to join). Otherwise clients need to either clone the git repo. or query github pages.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Thanks for the initial work and feedback msteiger :)

I just majorly updated the design above. Comments:
  • New "Distros" section for helping packaging together different distributions in Jenkins outside of the engine build job
  • New section with ideas on backwards compatibility
  • Big update of the workflow section and narrowing in on a better design (a single Index updater job called by module jobs). Key point: Expect execution to begin in Jenkins already aware of the module details for that run (will be passed in via Jenkins parameters - shouldn't need the module repo again)
  • If we put the scripts into the Index repo itself that helps a lot of different things (details above) - we can also have several different sets of scripts for different purposes (almost like TeraMisc)
  • GitHub probably won't be bothered until we've got either thousands of modules releasing left and right or thousands of users auto-downloading released artifacts from GitHub. We need a focus on future portability if we need to move to something different later (custom solution in AWS or what not - but anything of that level starts to require financial stability of some sort). Key point: Only module releases will upload to GitHub, not snapshots, so we're probably talking a few jar/zip files per week for the foreseeable future (and that's after actually getting module releases working). Downloads will be greater, but our traffic as-is remains minimal. Plenty of projects with actual traffic host files on GitHub just fine :)
  • Can use a convention for uploads to pre-determine download URL based on GitHub base URL + module name + release section + versioned jar/zip. This can be calculated in Jenkins during the Index update and can take advantage of any config in the repo itself
  • In-house hosted webserver might be good for finding multiplayer games, but anything we can throw at GitHub for now is a lot more traffic-safe (thus why Artifactory is for development and GitHub for release)
  • Not entirely sure how the Tracker project would pull its info. Use an embedded Git lib, download a src zip from GitHub, or web-query individual pages. Open to ideas!
 

msteiger

Active Member
Contributor
World
Architecture
Logistics
For now, I plan to use jgit in the Tracker, so we can use the github site directly, but it should be general enough so that it *could* connect to the game-master-server and leech the information directly from there without the git overhead.

So, I put the gradle script into the ModuleIndex git repo. right?
Cervator: can we setup Jenkins so that is run the script for all existing modules? And can you setup a repo. for the tracker?

btw: Thumbnails for modules would be a nice feature (maybe even auto-generated identicons for those who don't provide an icon/thumbnail image)
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Created MovingBlocks/ModuleManager and added access - uncertain of the name though. The function is akin to a package manager, just for modules. Not really sure where I got "Tracker" from in the first place - more of a site inspiration I think. Might rename it - any ideas?

Immortius - curious in particular if you have any high-level recommendations here, or if it all seems decently well put together. Should the manager & index be part of gestalt? gestalt-index and gestalt-manager?

msteiger - yep just put the Gradle script into Terasology/Index somewhere. I suggest initializing both the Index and Manager repos as Terasology "lib" style projects (check utility.gradle) so they can be fetched that way, akin to Jitter & friends. Although the Index may not be an actual Java project, just a bunch of scripts ..

For the Index you might want to use some top level directories before diving into individual module data dirs and script bases

Agreed on thumbnails/identicons, wonder if we could get a clever identicon that also works as a QR code with the module's URL associated with it somehow? Install a module on your server by scanning a code :3 Or we could just cheat and use the module name with GitHub's identicon preview to generate one (although unsure how well APIed that is)

As useful I'm happy to dig into this too if I can ever just find some time. Today (Sunday) started for reals around 11:55 pm after everything daily-day was out of the way, sigh :)
 

msteiger

Active Member
Contributor
World
Architecture
Logistics
Will look into it asap.

About the identicons: github creates them only for member names, but their algorithm seems to be fairly easy:

Our Identicons are simple 5x5 "pixel" sprites that are generated using a hash of the user's ID. The algorithm walks through the hash and turns pixels on or off depending on even or odd values. These generated patterns, combined with hash-determined color values, ensures a huge number of unique Identicons.
https://github.com/blog/1586-identicons

I use this algorithm here in other projects (they look similar to those on stackoverflow):
https://github.com/donpark/identicon
 

msteiger

Active Member
Contributor
World
Architecture
Logistics
I've written a python script that parses Github repos based on their API (v3) and creates a simple json file that contains all module.txt files from all modules.
The file is run directly by Jenkins and also archived there. The build is triggered by module builds (see the module buiild template).

There's already a view for the launcher that shows all mods, but proper download and install mechanisms are still required.

I think we can delete/move to graveyard organization the following repos: Terasology/Index MovingBlocks/ModuleManager and Terasology/FacadeAppletOLD (not a module)

@Cervator: what do you think?
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Belated reply here, we talked on IRC already. But just for the record:
  • FacadeAppletOLD - deleted
  • ModuleManager - still needed, intended to be the home for the download/install mechanism, both for launcher and game (in case somebody doesn't want to use the launcher), maybe even usable on a website (method to generate/update the module info to list)
  • Index - still needed, doing some other stuff in there too like distros and intending for it (or a related repo) to host another GitHub page with a minimal module site. We might also want to have the actual indexer script under source control sometime when we take start taking advantages of more stuff from module.txt :)
Finally got a chance to take an actual look at the job in Jenkins at least - cool stuff! Glad to see how relatively easy it looks to get stuff from GitHub.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Bump - got working Distros in Jenkins. Now the question is exactly how we want to organize that stuff, especially for the launcher, and how to change reading needed info as that's in multiple jobs now
I'm thinking we should add a new layer to what the user can pick. Currently you just pick your "Type" (stable, develop, legacy) and "Version" (build number), both under options. "Distro" could be a new top level option, maybe pickable on the main menu? It wouldn't have a lot yet, but in the future I could see it become the "Mod Pack" choice you have in existing MC launchers.
  • Bare (engine zip) - internal option - not a choice the user can use for playing, as without Core (eventually will disappear) you wouldn't be able to actually create a world. But it might be a base option the user could then customize modules on top (probably better to base on top of Iota in most cases though)
  • Iota - my new suggestion name for the old "Mode Zero" concept. Idea is it is the slightest amount of modules you'd need to meet your expectations of what to have in a typical game: A player, an inventory system, breathing, hunger, combat, a basic world, etc. For us those are modules. For most games they're the base game you then mod on top of.
    • This is special in another way as every included module should be community owned and maintained, including compatibility testing with every Iota module active together. We should guarantee this set as the "safe" batch of modules you can assume will be available for you to mod on top of. This also makes it the ideal platform to run automated acceptance tests against should we get those in the future.
    • Note that you could still be pick more restrictive gameplay templates within the Iota Distro. For instance there could be a free style setup that leaves combat and other stuff disabled
  • Omega - this is the old "all stable modules" lineup which is going to continuously grow bigger and less stable due to the sheer amount of modules. Stability with all Omega modules active at once would not be guaranteed ;) We'd probably still leave this as the default for the foreseeable future (effectively all the existing options are "Omega" distros)
  • AWT - our 2D version of the game with suitable modules. When it works again, anyway. Torn on whether you might want an AWT-Iota and AWT-Omega but I expect not all stable module would work well this way.
  • Some alternative mod pack not yet developed
  • Applet - not relevant for the launcher - could be using a minified module list / game version for faster download on the web
Beyond that choice the user could still go pick between stable, develop, legacy, or experimental (should we need that some day) - if that option is available for the Distro

Each Distro is currently defined simply via module list in a gradle.properties under https://github.com/Nanoware/Index/tree/master/distros (haven't merged to MovingBlocks yet) - that could be expanded with more Distro metadata if needed. That gives us the interesting option to have the Launcher load the Index instead of its internal configuration, before it ever checks Jenkins, and with stable release artifacts on GitHub even download the whole game without touching Jenkins (if it is down or otherwise unavailable / too slow).

I suspect we could even work in a whole mirroring system where the Index points out alternative download locations. For instance builds from February or older right now only exist on a backup machine at my house, with Jenkins getting too clogged up if it contains more than a few hundred builds per job. If instead we were to put artifacts + changelogs on a plain webserver anywhere (no Jenkins needed) that the Launcher could read we could easily have an older build archive or take advantage of available servers with space and bandwidth.

The Index really becomes our central database / squid to help organize everything - even with all our sites entirely unavailable. The more stuff we have on GitHub the safer we are, whether faced with traffic spikes, malicious actors, or even the loss of a central contributor (I consider "hit by a bus" insurance important for the project and aim to put more of my personal notes & access online as I'm able to). Another angle, which is out of scope right now, is maintaining module compatibility over time if a module's origin or name changes (ideal place to put information on that would be in the Index)

Some additional technical notes:
  • Distro build jobs can be run after the engine build, even after engine build and all module rebuilds (think full snapshot set of engine + all stable snapshot modules after rebuild)
  • Can repackage the game zip without rerunning the engine build, just to fix some modules. Question is how this might get versioned? Engine version didn't change.
  • With the lineup in Git we can easily reference it from multiple places and automatically update it (make @Gooey do it) without causing noise in the engine or module repos.
  • Distros could actually go fetch changelogs for all modules referenced and present that in the Launcher along with the engine changelog
  • You could make a "virtual" module for Iota at build-time that would define a gameplay option with soft dependencies (when we have that - defaults modules to on, but allows disabling) on all Iota modules (even in an Omega distro since Omega is a superset of Iota)
  • You could build a new gameplay module on top of Iota, without having to update individual dependencies (mainly just to have all the "base systems" live in the world)
I think we could come up with a fairly quick solution for the Launcher to load some basic info out of the Index (not defined yet) to replace the info in the GameJob enum, simply telling it to load the existing Jenkins setup from both the engine job (version info, changelog) + Omega distro job (game zip). Then behind the scenes we could build more options like the new distro selection, support for reading from GitHub releases, read from a plain webserver with a specified layout, eventually add in changelogs for modules, then finally outright module selection.

Getting around 2 am here and I might not have thought all this through fully yet, so please review and get me some feedback please :)
 
Top