msteiger is already interested in playing with it some. Feedback appreciated! As usual this is a proposal up for debate, not a design written in stone.
Summary
Use a single GitHub repo as an "Index" for available modules then have a library project to query said repo as a sort of repository database (like yum) to display available modules to a user in various scenarios (Site, Launcher, Game).
Index repo is expendable - it mirrors information available in Jenkins (released module jars) and in the GitHub module repos, but is a single central spot to query rather than iterate over all module repos under a specific GitHub organization (or worse: hitting all module jobs in our Jenkins or jars in Artifactory every time).
Offers a framework extensible by third parties who can create their own index type repos and have users hook up to those in addition to / instead of the official repo via configuration in launcher or in-game. Probably also reusable independently along with gestalt-module (gestalt-index?)
The Index
This is a single special repository under the GitHub Terasology organization (where all official modules live). In it each released module gets a module manifest similar to the module metadata file (module.txt/info), but with added per-release information like version number, jar file download link, dependencies as of that version, and so on. Similar to a GitHub-based version of our Artifactory, really, but backed by GitHub's infrastructure.
I figure a one JSON file per module approach should work. It'll only be updated when a release is promoted in Jenkins, not for every snapshot build, so updates shouldn't be super frequent (at least not without a massive amount of modules). Alternatively maybe each module gets its own directory with a single file for global stats + one file per release. Not sure what is preferred / more efficient.
The Index itself should be portable so it could live and be queried somewhere other than GitHub. It should define a format others can replicate to create competing index repos. Users could then add those in as alternative sources for modules (with an appropriate disclaimer about using third party modules that haven't gone through official channels)
The Site
An extremely basic module tracker site will live as a GitHub Page, no rating, commenting, or anything fancy available there, just manual browsing/downloads offered. This leaves space for a more feature-heavy custom site hosted elsewhere, yet even in the case of a traffic spike overwhelming the fancy site the basic site (and Index functionality) will remain available.
I am looking to the Netflix OSS index page for design inspiration here as it does per-repo stats and looks awesome. Imagine that, but for modules. I haven't dug into their setup enough to know if they are pulling repo stats live to build the page or if they also "cache" the stats somewhere first. While pulling stats like for 10-20 repos live would probably be doable I doubt it is for hundreds or even thousands of repos.
There should be a way to hide, dim out, or put disclaimers on modules that go inactive or break vs latest engine version. Or maybe highlighting version info would be enough, can then gradually fade entries if they haven't had new releases as new engine releases come out. The Index repo probably can know a little extra about the engine (or rather, the jobs in Jenkins that do the updates will know). Like the Index it should be possible to generate and host the site off GitHub if needed.
Probably the site should live in a second special repo under the Terasology org since it needs to be named something special, that way the Index repo also remains as small as possible. Yet it really needs all the info in the Index to be able to build the site, so I am not sure. Maybe it just contains the logic and then loads data from the Index repo.
The Tracker
A separate library project should be created under MovingBlocks on GitHub that can pull down and scan the Index repo. Downloading the whole thing as plain text (or as zip, really) shouldn't be awful, the repo should never have graphics or anything intense, at least not per-module. The tracker should then parse out all the release details in some sort of module data structure that can be interrogated by the calling software. Examples for this include:
- Official launcher. Add a new "modules" tab of some sort and allow pre-loading a set of them. This'll become more useful with more advanced launcher features, like automatically setting up a headless server including modules + config (I bet you could even set up a remote server via SSH!). There could also be a separate "News" section specific to module releases.
- Terasology itself. Likewise allow a list of available modules to be shown somewhere and be downloaded before creating a world. Could perhaps also be used server-side when it needs module updates before sending them to users after a version upgrade
- Fancier module tracking site. Akin to what Philaxx was working on - a full-fledged module site with reviews, comments, ratings, etc
Workflow
The Index will be maintained by Jenkins and depends on the full release management implementation for module build jobs. Ultimately that will result in a promotion approach to stable releases when a module author is ready to release. When a release build runs it'll trigger a series of extra steps not done for snapshots - some/all may be via Groovy script(s) rather than Gradle as it isn't exactly super flexible for when you want to do crazy things:
- Author triggers a promotion for module X via the normal "Unstable" build job (tied to develop branch)
- During module release promotion any needed version & doc updates are done and committed (develop -> master)
- "Stable" module job (tied to master) triggers on-commit with final release info
- Jar from master is published to Artifactory's normal release repo rather than the snapshot repo - only developers and other jobs will actually get files from here
- Jar will also upload to GitHub's "Releases" tab (there's an API for that) where users and module tracking will get files from
- Stable job may do some extras like updating a GitHub wiki page with doc updates, updating a Xenforo thread header with the latest module info, notifying for any "New release available!" needs, and so on
- On the stable job finishing the release from the master branch it'll trigger an Index update job as a post-build action, passing needed parameters. It'll also commit back to the develop branch (version update for next unstable + changes made in master) to trigger a new snapshot build
- Index update job is Git-configured to the Index repo rather than individual module repos. Will modify the appropriate module's manifest in the Index, then commit and git-push an update
- Unsure if this would also need to trigger an update for the site repo, if it isn't the same repo or loads fresh data from the Index repo each view
Most likely there will only be a single job in Jenkins responsible for updating the Index, pointing to the Index repo via Git, and triggered by the module jobs responsible for doing releases. It won't run on-commit, only indirectly. This gives us a nice spot to put related scripts and prevents multiple jobs from trying to update the Index at the same time (instead they'll queue up separate executions of the Index updater job). Parameters can be passed from the module release job, could even include module.txt or its contents (after release edits are applied)
Another related activity in Jenkins is post-engine release module tests. If an engine release runs it would be good to know if all the currently released modules work at that version, as well as the very latest module snapshot.
- On engine snapshot build: Trigger all module snapshot jobs to see if they still compile so authors will know early if upcoming engine changes break their stuff (already a goal)
- On engine release build: Test or do something with the released modules to see if they still work at their latest release or even a few versions back with the newly released version of the engine (new idea). Looking to somehow create a compatibility matrix (or other "decoration") in the Index repo, but this is probably a luxury item for later (beta level?), like when we can run some sort of integration tests separately against existing artifacts.
One of the "meta" things we can do in an Index repo is define a few recipes of engine + some set of modules == some flavored distribution of the game. Currently we just have a manually maintained list in Jenkins' engine build job, which needs to be split off to a later phase and made less hard coded.
I figure this can be done by maintaining a few lists in the Index repo, possibly with functionality advanced enough to exclude a module or a module set if build errors occur on release (might mainly be a problem after engine releases causes module rebuilds). It could also be used to offer a lighter bundle for the applet, or a distribution of the AWT mode with just a few relevant modules. Still wondering if this is a good idea or not. Potential distros:
- Bare game: just engine + facade, no modules, can't create a world (none available) - mainly for the Launcher to use as a base game download to then stack modules on top via user choice
- Mode Zero: engine + facade + MZ modules - the set of modules we consider part of the "core" game platform (inventory, basic world, basic player, basic creature AI, etc) - can create a simple world with basic gameplay that is the primary foundation to build more advanced modules on top of
- Applet: engine + facade + minimal modules to showcase the game. Would likely lack most the soundtrack to keep the size small, but highlight a few of the more interesting bits of content like Wood & Stone, Light & Shadow, Cities, etc
- AWT: engine + facade + few AWT-friendly modules. Terasology-lite / DF style with less intensive need for video card performance
- Fat client: engine + facade + ALL stable modules. Essentially the current game download zip.
This I think would be a huge help in defining distribution jobs in Jenkins, which has caused me some design headaches in the past.
Backwards Compatibility
Another utility option in the Index repo would be to change URLs, module names, etc in the metadata file there to maintain compatibility over time (since you could make that change as-of a particular release)
Say a module gets renamed from X to Y, on GitHub the module repo just gets renamed as does any affected files in it. Unless we leave behind a "shell" repo with a forward any old references would break. If a new module is ever released with the name of a different old module - all hell could break loose.
This (or some of it) may be better tracked in the actual module metadata file (module.txt/info) but in the Index this could include changed download locations and such as well, per release.
This is a lower priority piece of the greater picture. I'm also not sure how you would trigger a "rename" - if there even would be anything to trigger, maybe just updating the Index with the next release looking different would be enough combined with smart module scanning.
I also wonder if this might be helpful in maintaining backwards compatibility through world changes, but that's pretty much just speculation at this point
Related:
- Module categories - for decoration / display
- Release management - need to only run stuff on releases, not snapshots
- Expansion of Gooey automation - mainly to allow promotion (and thus index update) on bot-request
- Gestalt-module - since we might be able to leave the setup extensible for use outside of Terasology (and maybe the Index work or Tracker need that level of knowledge about the module setup)
