Release Management & Feature Analysis

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Tossing these two together for some discussion as they somewhat relate. I'll try to keep it relatively brief to encourage discussion rather than overwhelm, but I'm still me ... :)

Will use section spoilers to cut down on initially perceived heft ...

Overview

Recently a brief discussion on IRC with Marcin Sciesinski and synopia made me wonder about our feature development process and the products, for the latter specifically about maturity levels / versioning. At the same time mkalb has been playing with version info for the launcher, changelogs, and so on. GitHub also just introduced Releases. Finally UberWaffe has also brought the use of MoSCoW analysis into the mix.

Marcin's point was that if he didn't feel like designing, just implementing, there isn't much to simply pick up for L&S. Even with MoSCoW applied a lot is left up to the imagination of the implementer. It is more for high level analysis and prioritization than explaining in detail how a gameplay feature should actually work - like a victory condition. Expanding on / formalizing our process a little brings me to the following
Feature Analysis

Idea is that features get analysis added only as needed (although we might need a little alpha/beta/rc tracking for versioning)
  • Features that are driven entirely by an invididual (like Synopia's pathfinding) don't really need MoSCoW analysis, at least not at first. Wouldn't necessarily help the implementation / matter for a while since the author knows exactly what to do (feedback & suggestions are still good)
  • Features that get suggested but don't have a motivated individual driving it or is still too vague to start on should have MoSCoW added to at least get a better overview. I hope UberWaffe will be able to keep doing this occasionally as an overall Analyst :)
  • Features that still don't have a dedicated implementer or remain vague can get the MoSCoW expanded by a related Game Designer - best current example is SuperSnark being top organizer for L&S (talked to him about this and he's OK with being "Game Design Lead" for L&S and fleshing out threads - Design badge added!)
Likely threads/features that start out with a single motivated individual could end up needing analysis anyway down the road as the feature gets more complex. That's very dependent on what the feature is, however, some simple stuff just completes more or less.
Development Maturity

A big advantage of the MoSCoW so far has been the prioritization (Must, Should, Could, Would) and furthermore splitting that into 3-4 phases. I think that would be good to aim for in general to better judge where we're at as well as to help version content. Good example here.
  • First bare-bones version: Pre-alpha ("Experimental" ? "x") purely for getting started and testing some contents, not playable yet. Really just a first goal post to aim for.
  • First playable version: Alpha ("a") with actual content that can be played through, still might contain holes. Should be tracked a little more carefully after this point
  • First releasable version: Beta / release candidate ("b" or "rc") that is polished enough for normal users. Release candidates may be spun off for acceptance of some sort.
  • Released: 1.0.0 (not a MoSCoW phase as much as a stage / milestone). After this point we're on the hook for full API docs and change tracking if appropriate.
Tossed in some release stuff in there. Individual letters are meant to go into a Semantic Versioning scheme for non-master branches. More on that later
Hierarchical Design

On top or prioritizing and phasing stuff better we've also got some trickiness organizing between the different scopes of the project - Terasology the game --> Game Modes --> (mods) --> Modules --> (sub-module level). Right now most the MoSCoW is hitting what will become individual modules plus the L&S game mode itself. Could formalize and connect that better yet:
  • Modules are currently our smallest pieces and their MoSCoW doesn't sub-divide (yet). They each (should!) get a thread in the Incubator forum with analysis applied (if needed/desired)
    • We might possibly hit modules big enough to consider splitting out pieces in the future. Independent Incubators are possible as long as they get linked well enough. Example: World Gen is a big topic and already has several poorly linked threads.
  • Game Modes are for binding a series of Modules together. We really could do this with the MoSCoW as well, providing a high level analysis in the Game Mode thread that talks of independent incubators as line items? A Game Designer Lead should be tagged for each game mode; they will be important milestones for overall game progress. If the Game Mode has a specific art style maybe an art lead could be tagged specific to that sub-project rather than for all the things (which hasn't worked so well yet)
    • Modules are ultimately meant to be smaller pieces than traditional "mods" - for example the "oreominions" module just contains some models meant to be used by "miniions" which is more "mods" sized. Dunno if that's another layer to the analysis/structure between Game Modes and Modules
  • The whole of Terasology the game really could itself go through MoSCoW as well, giving us a better idea where we are for our global alpha-beta-released status. Again line items would probably relate to other Incubators (some of which could be entire game modes). Alpha probably would depend primarily on L&S plus develop/multiplayer convergence including module restructuring.
On top of these hierarchies we also have general brainstorming. The L&S main thread in the art forum is the best example as it isn't specific to individual features or the overall game mode, it is sort of just a mixed bag, but one that is vastly bigger than a single Incubator can hold. Probably we should keep exact analysis out of such threads and just go nuts with ideas, importing those that need work into Incubators with analysis - maybe the Game Designer for that game mode should be responsible for initiating that / keeping an eye out :)

Finally engine work is sort of separate from all this. World Gen is sort of in that boat too at the moment - maybe something to treat in a similar fashion to a game mode with Nym Traveel
on point, but not exactly the same. Likewise for Immortius on multiplayer. I know Xenforo has a thread prefix addon we might want to use on Incubator threads to better mark them than the single "Scope" field in the header template
Version All The Things

Plenty of stuff above has referenced release levels which is something we haven't really done for some time now. I was arbitrarily assigning version numbers to stable branch builds in the past, but it had no meaning. mkalb in particular has paid attention to this topic lately related to use in the Launcher and its tracking of the changelog of the primary project. I wonder if he might be interested in stepping up to become a Release Manager of sorts, helping keep track of all the numbering, dependency tracking, and Jenkins/Gradle fun? :)

Semantic Versioning ("semver") seems a good convention to follow and we might want to sort out what to us matches the different aspects of it. My thoughts:
  • Version all the things separately when we split them out - got a big draft nearly ready for review and feedback there. Seems obviously needed anyway to track module dependencies and such.
  • Use the master/stable branches much more. Builds from there get natural version numbers (no pre-release tokens) and matching GitHub releases
  • Anything from a develop branch is a pre-release build and gets the semver pre-release token + pre-release checkbox on GitHub. The pre-release token matches the stage of the MoSCoW we're in (so "alpha", or "a" or "rc" and so on, possibly with a number tagged on)
    • Challenge: Experimental builds ("x") would sort as more mature as alpha/release candidates unless we find a better prefix. Alternatively the length of the prerelease token can be used too - experimental < alpha < beta < rc
    • I'm feeling tempted to use the Jenkins build number after the first pre-release token, i.e. 0.5.0-alpha.25. This makes me feel guilty as it seems to go against the semver convention saying build metadata should be at the end after a +
    • Exactly how to correspond Jenkins build jobs with pre-release tokens might need work. Although with the develop job taught somehow to use alpha/beta/rc intelligently maybe only one job will handle all that - could work. Experimental builds have been a separate build job using a different branch in the past, which could be enough.
  • Version 0.x.y is initial development and not expected to fully adhere to the semver standards for when to increment major/minor/patch. I figure the MoSCoW stages and the implementers opinion for what's a change substantial enough to increment "minor". Patch level again really could just be the Jenkins job/build number, but then I'm violating semver again and screwing up the definition of the pre-release token.
  • Version 1.x.y is for when something is stable enough to follow the semver conventions exactly so other developers and modules can rely on the version number alone to calculate dependencies and plan around API changes. Possibly we could change from using Jenkins build numbers at the patch scope to using semver properly - although really a patch is any change, which takes a Jenkins build ...
This needs more discussion, despite already being the biggest section ... :)

One tricky bit is that if we involve the Jenkins job build numbers we must only use it from one job per repo, and keep master/develop branches in sync. Otherwise a Terasology unstable (pre-release) build from develop would speed ahead in patch number vs a stable build from the master. Would have to ignore the master build number and simply take off the pre-release token when pushing changes forward from develop to master. Say Jenkins develop build 123 would create 0.5.123-alpha; push to master and that would become 0.5.123 (but then would lose the informative "alpha" status ... wah, this is hard)

Edit: I'm thinking more about how semver really wants the alpha/beta pre-release stuff to happen betweeen natural releases (1.0.1-alpha, 1.0.1-beta, then 1.0.1 and afterwards work begins on 1.0.2-alpha) not alongside natural releases (1.0.1-alpha, 1.0.2-alpha, then 1.0.2 - no formal 1.0.1 release) But the magic 0.y.x period really is entirely "pre-release", so I wonder if it what I'm outlining above makes sense, then at 1.y.x you swap to alpha/beta/rc being entirely between natural releases.
Organizing Teams

A final brief note here on teams, or the lack of coherent teams beyond just badging people left and right.
  • Focus heavily on Game Modes (L&S) as the biggest sub-divisions for teams rather than function (Art, Architecture, Design, etc)
  • Make project-wide roles less overwhelming by focusing on helping game mode teams rather than all the things at once
  • Use GitHub milestones and issues to organize primarily Game Mode development, like the current example (1, 2) with L&S
An examplesetup for overall as well as the L&S game mode and with a possibility of "Steam Fortress" for another one

  • Project wide (some existing, some new)
    • begla as overall Lead Dev and 3D wizard, able to help overall on various topics
    • Me doing the "juggle all the things" thing
    • Immortius as Lead Architect to also be able to provide global guidance
    • Skaldarnar in an overall Design role of some sort (project state type threads and similar global stuff)
    • mkalb as Release Manager. Sort of close to what Skaldarnar has been doing, but more infrastructure / version specific
    • UberWaffe as overall Analyst of some sort with magic MoSCoW fairy dust to sprinkle over all the threads of the land
  • Light & Shadow
    • SuperSnark for Game Designer to add extra detail where needed
    • glasz for Art style to keep it consistent within the game mode
  • Steam Fortress (in short focused on DF features)
Note: List doesn't contain some of the other roles we have, just primarily the ones involved in release & game mode design

Ooookay. Yeah that's still long. Sorry! Thoughts? Aiming to post one of these threads per day I'm off, hah!
 

Immortius

Lead Software Architect
Contributor
Architecture
GUI
Some thoughts:
  • One thing we need is semantic versioning of the modules themselves - particularly so that dependencies can require a specific major version/minor version, and handle having multiple versions of the same module installed. So a module might depend on 1.1.0 of a module, which would allow it to work with 1.1.4 or 1.2.0, but not 2.0.0.
  • A clarification is that game modes would exist within a module.
  • My current plan for the multiplayer branch is to rework the persistence layer to support unloading/reloading entities with their host-chunks (major breaking change for saved games), port in relevant changes from develop and then make it available for general development, followed by looking at the modding API (particularly sandboxing mods and game types) and otherwise being available to add and merge engine-level functionality as it is desired. That should allow us to start semantic versioning the core engine - until that point we won't really have a defined API. I'ld probably start moving non-engine functionality into a core module at this point too. There will still be many breaking changes going forward though.
  • I would recommend not tying semantic versioning to jenkins builds. Semantic versioning is for releases - builds should just be marked as snapshots of the next target release version (so 0.3.1-SNAPSHOT-OtherInfo?). Someone will need to choose when to do a formal release, and it is at that point they should review the changes and increment the semantic version. Jenkins has a release plugin that works pretty well for maven in this space, don't know if there is something similar for gradle.
 

glasz

Active Member
Contributor
Art
I'm not really sure what kind of usefull comment i could make. One thing that comes to my mind : sort modules features according to their reusability : wich ones are completely specific to a single module and wich ones can be shared to Terasology as a whole game engine project. Select the more reusable ones as priority.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Jenkins has tons of fun toys that relate:

https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View (supported by the release and pipeline plugins)
https://wiki.jenkins-ci.org/display/JENKINS/Artifactory+Plugin (if we set up an Artifactory instance, which is easy enough)
https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Artifactory+Plugin+-+Release+Management

Lots of support for Gradle and custom stuff too. I knew these plugins existed but haven't tried really digging into them before - there is a lot of stuff in all that. As much as I love tinkering with those sorts of toys unless I'm doing it at work (which I haven't for a long time - although there's some potential I'm getting back into it soon) I don't think I'd find the time to really figure out the ideal setup.

Probably not a lot of drawbacks to just setting up a full Artifactory instance (I've done it for testing at work in the past and Immortius is using it for his job too) but even if we didn't want to we can use Jenkins jobs themselves for it via another plugin. Still, probably just easier to use Artifactory.

On the other hand nothing likely integrates with GitHub releases yet, not sure it is even in their API so far. That would be a better place to store artifacts so our server isn't critical. Now if there was only a way to treat GitHub as an Artifactory instance...
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Artifactory instance is up: http://artifactory.movingblocks.net

Bottom two links above are likely handy to look into how it could be put to use. We can put the proper Slick dependency in there, TeraBullet, Jitter, etc, as well as even the individually compiled Terasology modules

Let me know if somebody wants a user to play with it.

Edit: Also updated to the latest version of Jenkins, all existing plugins, and added the Release, Promoted Builds, and Artifactory plugins.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Can I have an Artifactory user?
Done :)

Edit: Oh, small discovery - Artifactory usernames appear to be capital sensitive
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Update! After a pile of Great Convergence work with restructure working with multiplayer + shiny new module versioning started by Immortius I want to set up in Jenkins and fully follow SemVer with two different promotion flows and two different jobs per module. I haven't implemented this, but after much discussion in the past, research, and brainstorming, this is my proposal :)
  • Job 1 = develop branch build, triggered by developer commits, always makes SNAPSHOT builds
  • Job 2 = master branch build, triggered by system commits caused by promotion in flow 1, always makes RELEASE builds
  • Promotion flow 1 = release promotion. Triggered from the develop job when the developer is satisfied and wants to release a new version
  • Promotion flow 2 = pre-release promotion (somewhat optional / lower priority). Triggered from the develop job when the developer wants to publish an early candidate
Most the magic resides in promotion flow 1. In a nutshell it performs the following:
  1. Dev picks a promotion parameter: major, minor, or patch (this is done on the develop job page in Jenkins)
  2. Promotion task in build script tweaks a few things
    1. Increments the version as per parameter (for instance 1.0.2 to 2.0.0) in module.txt, so a change to the codebase
      • Exception: If the pre-release flag is set the version has already been picked previously. Either adopt that or allow an "override" parameter to force a different version number
    2. Clears any set pre-release flag in module.txt (more on that later) - so a change to the codebase
    3. Does any automated housekeeping we might add later, like updating a changelog, parsing out details for a wiki page, etc. This might also cause a change in the codebase
  3. Commit locally
  4. Push to master branch (including the local commit and any commits since last release)
    1. This triggers a build of the master branch. This will be a RELEASE build of the version just promoted as per parameter (so for instance 2.0.0)
    2. This build will be published to Artifactory and GitHub as a release build
  5. Increment the version by a minimal patch increase to be ready for next release, so +0.0.1
  6. Commit locally
  7. Push to develop branch
    1. This triggers a build of the develop branch. This will be a SNAPSHOT build of the minimal next version (so for instance 2.0.1-SNAPSHOT)
Normal builds make no changes to the codebase, so there's no commit. That would otherwise result in circular builds without hacks to prevent that. Instead they purely produce SNAPSHOT builds for the next minimally distant release (at least a patch release, thus the +0.0.1). If we want unique jar file names for these builds (not intending to push them to Artifactory/GitHub) we can add +[jenkins job build number] such as 2.0.1-SNAPSHOT+123. We could even edit that into the built-jar, if wanted, without committing the change, but I'm not sure that's worth it (or where to put it - a snapshot field in module.txt?)

The pre-release flow differs slightly and is somewhat optional - might not be needed until later when more stuff is at or past 1.0.0 and serious developers with significant userbases want to publish early candidate builds after a major version increase in the engine or so.
  1. Dev picks the pre-release promotion flow with parameter: alpha, beta, rc (again done on the develop job page in Jenkins) as well as the version parameter for the to-be release
  2. Promotion task runs
    1. Increment the version accordingly
    2. Check the pre-release flag in module.txt
      • If it is not set previously then set with the parameter name and add .1 (so alpha.1 for example)
      • If it is set previously and matches the parameter then increment the number (so alpha.2)
      • If it is set previously and doesn't match the parameter then replace with new name and start at 1 (so beta.1)
    3. Edit the already-built jar file and name it using both the version number plus the release tag, such as 2.0.0-alpha.1. Publish it to Artifactory/GitHub with the pre-release flag (on those services) set
  3. Commit locally
  4. Push to develop branch
    1. This triggers a a build of the develop branch. This will be a normal SNAPSHOT build but it will happen to match the pre-release build exactly. So the published 2.0.0-alpha.1 jar might equal 2.0.0-SNAPSHOT+123 built jar.
I figure "duplicate" builds like that are okay in this one case (release builds conversely differ by the +0.0.1 version increment). I think it will sort correctly as per SemVer convention due to "alpha.1" having two pre-release tokens which beats "SNAPSHOT+123" which has one pre-release token and one build metadata token (which is effectively ignored)

The thing I struggled with the most in this whole mess is how to deal with almost needing a psychic prediction of whether the next release is going to be a patch, minor, or major release at the time you start it (to have an accurate new version number to base off). By incrementing +0.0.1 after a release that's the minimally possible next release and by the time the dev is ready to publish a pre-release build the content is probably known well enough to pick patch/minor/major. That is why that promotion flow allows the choice and commits the version number back for use in the future. The eventual release promotion either accepts that choice (a "respect pre-release version" choice parameter to go along with patch/minor/major?) or overrides it.

Weakness: Does not support promoting a release build that isn't the latest, and may be vulnerable to commits made during the brief release promotion activity window. An alternative to that would be making the release promotion publish the existing built jar (possibly modifying it directly first) rather than push to master and letting that build do the publish of the release. That would allow promoting older builds but could still bump into other commits when updating, and may not make logical sense / happen often enough to complicate the setup further.

Modules would be born at release 0.1.0 and increment normally from there, although technically < 1.0.0 releases follow the rules more loosely. Developer can choose to promote the very first build to 1.0.0, that's up to them.

Patch-level release promotions would not actually increment the version number since that was the minimal increase in the last normal release (after releasing 1.0.0 you're already working on 1.0.1 so don't need to increment again). Exception: Developer promotes the very first release at patch level (there was no previous release to do the +0.0.1 increment) - need to force an increment there from 0.1.0 to 0.1.1

Thoughts? I know this is long as usual, but I would like to think it is also pretty darn comprehensive! :D
 

Immortius

Lead Software Architect
Contributor
Architecture
GUI
This all sounds sensible. I don't think the lack of promoting a release build that isn't the latest is an issue, and I assume that whatever distribution site we end up using will allow manual uploads anyway which will doing non-standard releases if necessary (such as releasing a patch to an older version or what-ever).
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Good note on patch to older version, that one slipped by me.

For that I suspect we'd have to do a patch branch off an older git tag (the promotion process should probably use git tags), then use a one-off patch build job. Pretty sure that's doable, but not something we need to worry about too much until we need it :)

Thanks!
 
Top