Development methodology and hi students from Porto

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
So I got a note (several really, and maybe others did too) from Porto, Portugal, where some students are doing a project seemingly analyzing open source projects for development methodology and such. Terasology has come up with some questions asked :)

  • What kind of process the team uses in the development of the project( Waterfall, XP, Scrum...) and why?
  • Also what do you think are the main advantages and disadvantages of following that process?
  • Who manages the team and sets objectives?
  • How often do you need to put out deliveries, releases?
  • Are there deadlines/milestones proposed to the developers?
  • How do you (the team) deal with issues and bugs (how you discover them, how do you work on them, when do you work on them)?
  • How do you deal with pull requests?
I thought I'd get a forum thread up for discussion also as maybe more than one group ended up picking Terasology. On top of that this is a topic dear to my heart that has come up from time to time and others might have some valuable input as well.

It is good timing as well as I've been wanting to go over the topic again and try to help us get more focused as a team. So maybe we can take this opportunity to not just mention what we're currently doing, but what we could be doing, and what probably isn't suitable for us but may still be very useful for others.

The main challenge

We're an open source, all-volunteer, non-profit project. We repeatedly have been near the sort of "critical mass" needed to have enough activity to attract more contributors to create more activity to attract more contributors and so on, but haven't quite gotten past it permanently yet. Most our long-term contributors have day jobs keeping us busy, sometimes too busy for long periods of time.

This makes following traditional processes from a work setting difficult. Methodologies like Waterfall, XP, Scrum, and any kind of structured planning with objectives, deadlines, and so on is nearly impossible to apply. In agile terms the "focus factor" for us as an overall team is exceedingly low, probably in the single digits (0-9%). We simply never know who will be available, when, for how long.

In a work setting where you're employed x hours a week that is vastly easier to deal with. You also get paid to do the work, which tends to help :D

Some larger open source projects (or those that are actually commercial ventures with open source components) can manage to organize better as they're far past the "critical mass" point (often by having paid staff involved or a direct monetary incentive). There is enough momentum and time availability to better predict schedules. In other words their "focus factor" is higher.

Questions

Will try to answer the questions to the best of my ability anyway, mixing in some experience from work and some future potential where able. I also wrote a related thread with some ideas quite a while back.
  • Development process: Nothing formal. This is the challenge listed above. Pretty much we maintain our issue tracker for identified issues / needed improvements, the suggestion forum for ideas, and several more dev forums here to shepherd suggestions through implementation into maintenance. Usually this happens whenever a new contributor shows up and wants to do something. The ideal setup would go something like:
    • Somebody makes a suggestion in that forum. Great idea!
    • Somebody takes it on for initial design and writes up more technical notes and details on how to implement it
    • Somebody starts coding it - might be the same person and often this is the point where the process actually starts (skipping the suggestion phase because a person showed up excited to do something specific)
    • We move the concept into the module forum if it fits into a module (most content does), into the art forum if it is to make assets for the game instead of code, or the Core Projects forum if it is for architecture supporting the game engine itself or other library-level functionality (not content)
    • The new feature becomes sufficiently complete to be put to use. Modules all live in their own repos on GitHub so authors often push directly without pull requests (PRs). Engine features hit the PR stage, get reviewed, and merged
    • We release the new feature in the next game release.
  • As for the why to our process: well, it gives what little structure to the process we can apply without getting in the way of people wanting to do work :) As noted often it gets short-cut when somebody is excited about a feature and shows up with it out of nowhere. Often the first time we as a community learn about a nifty new feature or piece of content is when an author announces it or sends us a PR. Hard to get very structured that way. Naturally this makes planning and estimating anything extremely difficult.
  • The team/teams are entirely self-managed by individuals. Everybody sets their own pace that fits how busy they are in RL. This is a huge disadvantage vs being able to treat something as a full-time job. I'm project lead by title but really do nothing but encourage and sometimes badger people to get stuff done. I step in where I can when I have time to do stuff that needs to be done that nobody else is getting to. Being super busy I unfortunately fall short in this constantly.
  • I aim to get a release out every 2-4 weeks, but this is highly erratic and depends almost entirely on who gets what done. Is there enough stuff to really do a new game release? Our game releases come with a large list of modules considered "stable" and included with the base game, but are all maintained by different people so it can sometimes be months between updates there.
  • No deadlines or formal milestones. We sometimes try to do a discussion on what's most needed at some given time, and with a bit of luck some bits of what we need gets done over the next few months. For instance a few times we've tried to come up with a todo list for hitting Alpha-level readiness. Beyond Terasology's primary game component we've used GitHub milestones with some success for our Launcher and new project Destination Sol, both of which are much smaller projects on their own, making them a bit easier to handle.
  • As for bugs anybody can report them on GitHub. Occasionally I'll try to badger somebody specific to see if they can check out and maybe fix a bug, but usually issues just sit there until somebody spots something they want to work on. Both external people new to the project (mainly players) report bugs as well as existing community members while developing and testing content.
  • Ideally pull requests get reviewed by at least one person other than the author who is familiar with the area impacted. That person grabs the code and tests it locally to make sure it works and does what is advertised. Then either reports back on the PR or merges it. Often PRs are done for code where the author is the only person really working in that area in which case I'll either try to personally test or merge it or find somebody else more familiar than me who can check it out.
Beyond Terasology I'm trying to improve the SDLC at the place where I work. So I'm reviewing different methodologies all the time and am pushing both Agile and DevOps a lot, both for tool sets and culture. I've taken a particular liking to Behavior Driven Development, which is sort of Test Driven Development taken one step further. Unlike the big team structured methodologies (waterfall, agile, etc) applying these kinds of techniques to Terasology seems like it might have potential, if we could just get the basics in place. More automated testing for instance is a no-brainer, it just needs work. I wrote up a big suggestion on the topic with much more detail.

One particular important topic is versioning. We try to use Semantic Versioning, which makes dealing with many small interconnected parts easier, as you should be able to tell the scope of a changed library based on its version number (and automatically pick up some minor change, but not major ones, without changing your dependency configuration). One idea that has come up recently is pushing our engine to v1.0.0 to declare it sufficiently ready for "production" usage, yet still keep the overall game in Alpha (upgraded from pre-alpha). Then stick to the "contract" of having the engine remain fully compatible in the future until such a date we do a new major release (v2.0.0). That in theory should make it easier for modders to create and maintain stable content.

More student / course stuff

Finally, for the students, what other questions may we be able to help with? I got a PDF with a course outline but am still not entirely sure. For instance aspect-oriented programming is on there which is oddly specific if it is about overall project organizing.

We don't use AOP directly although in a sense our entity system is similar. AOP uses cross-cutting aspects (common code) applied dynamically to specific parts of a project in need of it. The Spring Framework for Java webapps is a great example heavily using AOP, and I recently started using jcabi-aspects in my main Spring webapp for my day job to add some nice utility.

Our entity system applies components (data containers) to entities giving them functionality via common code in systems. So a creature would have a model component giving it shape, a health component giving it life, and so on. During game execution the main loop goes through all the systems and each runs the logic on entities with relevant components (so entities with a model get rendered)

Fun fact: AOP as a programming paradigm is patented. I had no idea you could even patent something like that. It made me worry for a bit applying more of it to my main work app - until I learned that the company I work for, Xerox, is actually who patented it, heh. Not like it seems we're doing anything with said patent. Xerox is terrible at monetizing stuff, seemingly just like Sun Microsystem was.

Anybody else feel free to chime in, ask more questions, comment on the above, propose other suggestions and so on. If anybody actually manages to read this whole thing. I swear I was just trying to write something real quick!
 
Last edited:

andrelago13

New Member
Contributor
Hi! I'm working with Aniiram on your project and your message was really helpful to us! You gave us even more information than we expected so we have everything we need so far. Right now we are studying how requirements are managed along the project.

From what you wrote we understood that anyone can suggest new features using the suggestions forum you mentioned. From there, the original author of the suggestion or someone interested can dig deeper into it, technically specifying what needs to be done to implement the suggested functionality. The next step is moving the idea to the module forum, therefore integrating it within one of the already existing module repositories. When the code is implemented within the module repository and the new functionality is stable enough, it gets merged into the repository without the need for a pull request. Later, you (or someone highly responsible for the main Terasology repository) test the code to check that it really does what it's supposed to do, and add it to the next release of the game. Is this correct?

Also, how is it possible to make changes or add new functionality to "deeper stuff" such as the game engine or any other parts of the project that are not inside any of the modules? Is this done via the normal PR method?

Once again thank you very much for your help, it really is great to study a project like Terasology and get in contact with other people dedicated to it!
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
@andrelago13 -very close, yep.

Modules are all stored in independent repositories and stay there. We build them in our Jenkins server and distribute from there. Most modules have just one author so PRs are rarely needed, but occasionally in a more active module there might be multiple authors, some review, and use of PRs.

The engine (and other core frameworks) tend to have multiple authors so PRs are typically used for extra review and how changes are merged.

Every few weeks when I've got some spare time I'll go through this gigantic local workspace where I've got the engine in source + 80 (and slowly rising) modules in source form and do some general testing, then publish a new release if all looks well. We have a stable lineup of modules we refer to as the "Omega Distribution"- the 80 modules you see in the main readme

Everything remains in their own repositories (and forks for different authors) but our engine repository has some extra "magic" via Gradle so you can easily fetch additional modules in either source or binary form and keep working in a single local workspace. For instance running "gradlew fetchModuleSample" retrieves the "Sample" module's source from GitHub and adds it to your engine workspace.

That beats having to deal with modules in separate local project directories and moving stuff around manually after making changes, yet allows us to stay focused on GitHub with activity only in affected repositories. Trying to keep all of Terasology including modules in a single repository would be nearly unworkable :)

All "core frameworks" including the engine live on GitHub under the https://github.com/MovingBlocks organization

All module repos live on GitHub under the https://github.com/Terasology/ organization

This is admittedly a little confusing since the engine repo is https://github.com/MovingBlocks/Terasology ... but that's a consequence of that originally being the only repo long ago. It would be more appropriately be named "engine" or so.

We nearly created a "MovingBlocks" module as well, to contain functionality about blocks that could move :D That would've been Terasology/MovingBlocks but we went with "MobileBlocks" instead ;)

Glad to help!
 

andrelago13

New Member
Contributor
Once again thank you @Cervator for your useful help!

Using independant repositories seems like a good strategy for your module system since you use Gradle to fetch them later.

I think it's also good to have new releases every 2-4 weeks because it keeps the community interested and helps people to better know the new functionalities added.

Naming the module you referred as "MovingBlocks" would have been quite a mess :) it was a good idea to rename it!

We'll get in touch with you if we have anything else to ask!
 

Francisco Veiga

New Member
Hello Cervator,
I would like to tank you for your help.
Also, i would like to ask you about the Architectural design of Terasology. Who do you structure this project and if there is any documentation used for this structures.
Do you have a Architectural pattern?
Once more thank you.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Hi @Francisco Veiga and no problem :)

Terasology originally started as a simple tech demo called Blockmania done by @begla to see if he could write the rendering engine for a Minecraft-like game. He does rendering work and 3D effects for a living. So that was the simple driver at the time - no deep long-term architecture that I know of.

When I then convinced him we could turn Blockmania into a game proper Terasology was born. We still didn't have much architectural guidance until @Immortius joined us and became lead architect. There is a fairly dated architectural vision thread here in the forum along with an assortment of other threads for code conventions and so on. There is also a wiki page about our entity system's architecture, which is really the backbone of the engine (an entity component system - lots of general hits for that on Google). And finally a codebase structure wiki page.

We occasionally touch on needed future architectural tweaks and have some written down informally like "Make camera system based on Components attached to Entities" - but that's really it.
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Hello Cervator!
Thank you!
And components? Packages?
Components are explained in the entity system architecture (they're data containers). Although maybe you mean component in a different way? Lots of different things get called components :)

Packages - I don't think we have a lot of convention when it comes to package tree structure for our classes. Just org.terasology.something and try using general Java best practices for sorting things.

We have an @Api annotation we can apply to a method, class, or whole package, to identify that code as approved for use by modules. Otherwise they're sandboxed for security (for instance: modules cannot access Files directly, but we expose ways in the engine to interact with stuff stored as files)
 

andrelago13

New Member
Contributor
Hello again @Cervator. We are now interested in studying how the software in Terasology is tested, both for the engine and individual modules. Dou you use any sort of automated tests or is stuff tested manually?

Also, at this stage we would like to contribute to the project by solving at least one issue reported in the GitHub repository. Would you advise us to start with an issue marked as "Contributor-friendly"?
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Sure :)

I'll roll general code metrics in with the automated testing. If you look at the main Terasology (engine) job in Jenkins you'll see a set of graphs on the right side. The bottom one shows the automated unit tests, written in JUnit, and they get executed every single engine build as well as automatically for pull requests submitted to GitHub. That way we know everything we have covered by unit tests should still work, although naturally there could be bugs in the tests themselves so there's no guarantee. But it beats not testing at all. It also tells us if a PR causes some unit tests to fail or other metrics to worsen.

Beyond unit tests we have several other metrics - in order of appearance in Jenkins:
  • Checkstyle - scans our code style / conventions to make sure the code is consistently using the same style everywhere
  • FindBugs - scans for some common code drawbacks that can easily lead to bugs
  • PMD - more of the same - style, common issues, etc
  • Open Tasks - simply looks for TODO tags in the code indicating something needs to be done
  • Static analysis - aggregate of the above (all them of added together)
  • Code coverage - how many lines of code are actually exercised by the unit tests
The same metrics are available for modules, although they're not always active or appropriate. Some modules just contain art so code metrics make no sense. Other modules just don't have any unit tests because the author hasn't added any (how comfortable different authors are with unit testing varies hugely). There's a statistics view also available in Jenkins that shows metrics across a large amount of our build jobs.

On a slightly more technical level you may have noticed there is both an engine and an engine-tests directory in the source code. Usually you keep them in the same project/module, just under src/main/java and src/test/java, but since we have some utility test classes for use by modules we had to split them out to get the dependency resolution right. Module unit tests depends on engine-tests depends on engine.

As for release type testing that's currently just a manual setup mainly done by me right before doing a release. I have a big workspace with every single module in the "Omega" lineup present in source form so I'll run the game several times with different gameplay templates and do some basic smoke testing to make sure the basics work. This doesn't cover all the modules though, especially those not in a gameplay template, and it doesn't go very deep at all. I only cover a minimal amount of the content in Throughout the Ages and JoshariasSurvival for instance. I have a big suggestion written up about how to improve this further with automated acceptance testing, but as with everything else we're short on effort to make it happen :)

As for solving an issue on GitHub: yes please! And you are correct, the Contributor Friendly issues list is ideal for checking for something easy. I'd suggest #1144 (copy existing code for new settings) or maybe #520 (display a new label in a corner of the main menu). But if any of you spot something more interesting and want a challenge feel free to go for any of the issues, just start small :)
 

andrelago13

New Member
Contributor
Why is there no line coverage or test results for most of the indicated modules in the Jenkings (http://jenkins.terasology.org/view/Statistics/) page you mentioned? Does this mean there are no tests for these modules or is there another explanation?

Also, how important do you think unit tests are for Terasology? Is it intended to test a very large portion of the code or just some smaller yet crucial parts of it?
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Why is there no line coverage or test results for most of the indicated modules in the Jenkings (http://jenkins.terasology.org/view/Statistics/) page you mentioned? Does this mean there are no tests for these modules or is there another explanation?

Also, how important do you think unit tests are for Terasology? Is it intended to test a very large portion of the code or just some smaller yet crucial parts of it?
Main reason for Jenkins not showing line coverage is that I simply haven't gotten around to enabling it everywhere :) We hooked it up to the engine, then a few test modules. There is an option to hook in a whole code coverage site for more details and to get updated stats posted to a PR (codecov.io and coveralls.io for instance). That's when I ran out of time last.

The goal for unit testing (displayed via the code coverage percentage) is for it to test a very large portion of the code, absolutely. But it is one thing to set it as a goal and another to achieve it :D

Beyond unit testing I want a bunch of other types of automated testing and metrics. There are several related threads and post, like this one, but they get pretty technical and complex, also digging into release management as a whole, not an easy topic! :)
 

andrelago13

New Member
Contributor
In this case, do you think unit/automated tests cover most or all of the important parts of the code or do you think you should dedicate some more time to do this to improve testing overall on Terasology?

Also, which do you think are the main external technologies used in the project? (if you use any)
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
In this case, do you think unit/automated tests cover most or all of the important parts of the code or do you think you should dedicate some more time to do this to improve testing overall on Terasology?

Also, which do you think are the main external technologies used in the project? (if you use any)
I favor more automated testing, just because it can be difficult to find availability to test manually on an on-going basis. With automated tests at least we only have to write them once, then fix if something breaks.

Main external tech - what do you mean? Like code libraries powering Terasology the game (main piece there is LWJGL) or software we use in our infrastructure like the forum here, Jenkins, etc?
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Merged the PR and added @gtugablue to the main game credits, let me know if others of you were involved in it too and I'll add some more names :)

I hope you'll be interested to keep contributing! That was a very nice fix
 

andrelago13

New Member
Contributor
Great!

About the libraries you use, do you think using external libraries makes testing any more difficult or easier? Does it impact the way automated or manual tests are done?
 

Cervator

Org Co-Founder & Project Lead
Contributor
Design
Logistics
SpecOps
Well, using external libraries in the first place is a great thing to do, so you don't reinvent the wheel and have less code to test in the first place. Of course you need to make sure each library itself is well maintained and tested. If that's the case it is an absolute win - you have less code to worry about. Plenty of libraries also directly aid testing (JUnit itself is a library, along with others) or indirectly aid testing by making mocks and other testing-related utilities available :)
 
Top