Architecture discussion for v3 [engine split / engine-libs]

Discussion in 'Developer Portal' started by oniatus, Aug 29, 2017.

  1. oniatus

    oniatus Member

    The v2 branch already includes a lot of features, most of them coming from the actual gsoc.
    I would like to use this thread for a discussion how the architecture of the engine should look like in v3.

    My main issue with the current project is the size. Everything is in the engine, all in one huge project.
    The advantage is obvious, you find everything in one place.
    The disadvantage in my opinion: Everything should not be in one place, especially not when building software :)

    We have a lot of subpackages, lets have a look at them:
    • audio: sound stuff
    • config: property system, to be extended/exchanged with the flexible config soon
    • context: dependency registry
    • engine: wiring systems together (oh, we have an engine in the engine - engineception?)
    • entitySystem: engine layer above gestalt entitysystem, adds lifecycle and prefab support
    • game: data classes for a game (setup, seed, ...)
    • i18n: internationalization
    • identity: certificate stuff
    • input: input and bindings management
    • logic: (ranting ;)) "i don't know where to fit my sources, but it is some kind of logic" - contains a bit of everything
    • math: engine math classes, based on TeraMath and bullet
    • monitoring: performance monitoring
    • network: network logic, also one of the "no one will ever touch it" sections :rolleyes:
    • particles: particle system
    • persistence: serialization of events/entities
    • physics: physics system
    • reflection: utility classes for reflection
    • registry: close to context, holds the dependency injection
    • rendering: magic section :)
    • telemetry: telemetry gsoc code
    • unicode: i foud no references to any class in this package, looks like dead constants to me
    • utilities: close to logic, a bit of everything
    • version: version dto
    • world: world stuff, like generator logic, block shapes etc.

    In my opinion, these should be separate projects and have as little dependencies to each other as possible. This way we can test them separate, define a proper api for each block/layer and wire everything together in the engine. In fact, we did that already, it is only hidden in one huge thing with 250k lines of code o_O

    Of course it will not make sense to do a hard package split, we need to inspect things in detail.
    Therefore it will make little sense to do it in one huge change.

    What I would propose is the following:
    1. Introduce a new layer between modules and the engine: Engine-libs. Move the entire engine project to engine-old and make it depend on the old engine. The code will do the same and be under the same versioning.
    2. Start to move small pieces to more engine-libs, e.g. math, physics, ... The code will stay the same and will also be under the same versioning (I think checking out 30 repos to get the engine working is not useful).
    This will also reveal architecture isses because the new engine-libs should not depend on engine-old.
    If we repeat this for some time, we may end up with a better split based on projects, have the option to throw away entire projects with a new implementation (inventory is next? :sneaky:) and make it easier for a new developer to get into the code.

    Opinions are welcome :gooey:
    Last edited: Aug 29, 2017
  2. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    Disclaimer: Everything IMHO and possibly sleep deprived rantings :)

    Good stuff and agreed :)

    I still have a v2 architecture/goal post floating around in my head to post sometime + set up related GitHub projects and such. I think the focus there should be finalizing the game logic foundation - the external API for modding and such. Multi-world, extracted Gestalt-entity, etc. The building pieces we'd want to have in place to go a long time without breaking changes, even something we could hit Beta with. That may be more of a v3 if we want to merge all the GSOC stuff needing a v2 and this could be v4, still a bit unsure ...

    This (continuing to call it v3 for now) could be the next round after that, which cleverly could be done in a way that doesn't affect the external API. So in that case it might even sneak by without needing a major version increment for the engine, but it might justify it anyway.

    I think there are three different categories of stuff to extract:

    1. (External) Libraries - things that have utility on their own, even perhaps to other projects. TeraMath is a good example, not sure how much is left in the engine or if the rest could be moved over. In that case we actually might want to move to JOML or something from LibGDX, maybe redoing TeraMath as a minimal wrapper with voxel-specific extensions. Context/registry might be another useful thing to extract (has its own thread). Entity system partially goes away when we depend on gestalt-entity instead (hoping for v2 - can dream, right?). Maybe even something like the renderer one day. World generation with facets? That might be an extension instead ...
    2. Modules - some things remain in the engine that could be moved to modules. Particle system might be a good example, and it may well need rendering to proceed a bit more to enhance what can be done in modules. The behavior tree stuff is another one (and is at least partially done with that GSOC item, again maybe pending more extraction/reorganizing - unsure)
    3. (Engine) Extensions (the engine-libs thing) - things that aren't really entirely complete on their own, and wouldn't make much sense as an independent library. This would be stuff that is tied to the engine so much and/or is an optional extension to the engine.Telemetry may be a good example, since the whole thing can be disabled. Maybe the performance monitor or some of the world gen stuff. Maybe even Soni's desired internet/database/filesystem connectors, but that feels more like highly privileged modules IMHO. We'd likely want to ship all these with the game but leave them optional and allow alternatives. Gradle/Groovy/Git magic can help develop them (and everything else) in a single workspace.
    It is late here and this is just a "real quick" initial response (yes, quick!), but we could go over the packages in more detail and thinking about them individually might help a lot. Like would i18n be an external library or a subsystem / engine lib / extension?

    As for approach I would be very hesitant to rename/move the entire engine dir in one go, as it tends to end up resulting in huge headaches around pending work (or a tricky code freeze) and handles poorly in revision history (at least some tools don't follow renamed/moved stuff well). I'd just start formalizing the modding API better (public API / front-end), letting it optionally start rerouting away from an "old" engine system to a new one at some point in the future, doing systems one by one. Eventually as activity moves to a new dir/repo the old history becomes less useful, the affected files can be marked as deprecated, and finally deleted in a major engine release some time after.
  3. manu3d

    manu3d Pixel Forge Artisan

    Generally speaking I agree with @Cervator's assessment. @oniatus I also agree with your highlighting of the problem, I'm just less convinced about drastic solutions.

    I'd certainly endorse a package-by-package evaluation and a roadmap tagging some packages for extraction from the engine. I don't think it would make sense that -all- packages listed above become separate projects, certainly not initially.

    I.e. it is possible to identify a number of packages that makes sense to keep in the engine as 1) they characterize the engine and differentiate it from other engines 2) they provide services that are very likely to be used in all Terasology-based projects 3) provide services that are not exactly exciting aspect in the videogame world. I.e. I am thinking about config, reflection, internationalization, telemetry and inputs packages to name a few.

    On the other hand I can certainly see some packages such as rendering, particles, logic and game that perhaps could be extracted - the first two for sure, the other ones I don't know them enough.

    So, there is potential there, it sounds quite an effort from where we are. I'd want to be well into v2 and perhaps even in an intermediate v3 before we tackle something as big as what you are suggesting.
  4. oniatus

    oniatus Member

    I don't want to say we should do all at once :D
    What we can do, even without touching the existing codebase:
    -create a new project engine2 (or a better name)
    -make PC and the other facades depend on engine2
    -make engine2 depend on engine
    -keep engine-tests as is

    engine2 would be in a folder next to engine and under the same version control and be initial empty. This will not break existing PRs and preserve existing functionality.
    -create a second folder engine-libs, containing fully qualified gradle projects.
    -once we identify a feature set which can be extracted, create a new engine-lib-project for it
    -add the engine-lib-project as dependency to engine2 and move the code from engine to the engine-lib-project
    -move tests for the feature directly to the feature project
    -(best case) add CI for each engine-lib-project, they should be buildable and testable on their own and have NO dependency to engine2, they are likely to require engine as dependency.

    We can always extract features with no pending PRs to keep conflicts as low as possible. If we iterate the process over a longer time period we may end with an empty engine, a lot of wiring logic (subsystems, main loop,...) in the engine2 and a lot of small projects with little to no cross-dependencies which can be tested or exchanged as needed.

    My goal is not to extract separate projects and repositories which can be re-used in other engines or third projects. Instead I want to split based on responsibility to clean up the architecture and grow clean islands in new projects where it is easier to get started than hopping in the entire engine at once.
  5. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    Okay, I think I understand you better now then. You want "engine-libs" to be like "facades" with subdirs being entirely independent sub-projects / modules (in IntelliJ speak, not Terasology speak - but perhaps also in Java 9 speak). That helps split everything apart and you more readily can calculate unit test coverage and solidify "sub-APIs" per component rather than let the engine be its own huge thing. I had just thought about the extensions concept for high privilege modules, but really the logistical benefits of splitting apart components/APIs for unit testing is a big deal.

    Not so much split out into independent repos, but split apart within the engine repo. Although at that point - why not have separate repos? Multiple CI jobs on the same repo (one building engine-libs/telemetry, another building engine-libs/particles etc) seem like they'd be kinda awkward (but doable). The PR builder type build logic tends to expect one "thing" per repo, so that any PR applies to the specific thing in that repo. I dunno how much hassle it would be to listen globally (some commit landed in the repo) but act locally (only do CI, review, etc, for the subset). Pretty sure you can do it, especially with Jenkinsfiles, but dunno how awkward it might get. That was the approach on my last work team and IMHO it was awful - but admittedly those were mega-monolith legacy apps facing a flawed DevOps design (again IMHO - DevOps is so vague these days)

    Whether we find ways to treat them as clean islands within a single repo or split out into repos but Gradle/Groovy logic it so well it becomes seamless I agree with that part of the approach :)

    I'm still not convinced about "engine2" tho. What does that buy us that moving, say, telemetry under engine-libs/ doesn't on its own? It might feel cleaner to do the roundabout step of fitting in engine2 as an API layer of sorts? Which is what I was getting at a bit last post: formalize the modding API better. Maybe modding API is too low scope there - a new API layer so we have engine - API - engine-libs - API - modules? Or something like that. Then we also don't need to rename "engine2" to anything more sensible later - just call it something to do with API layering?

    In any case we certainly could start by simply splitting things out of the engine dir, but staying within the same repo. Much like how we took content-type stuff out of the engine and parked it in Core, pending full extraction out into purpose-specific module repos.

    I'm very split repo happy (since I like CI and PRs per repo), but will admit it adds to the logistics, without supremely well done automation. And it is largely on me not having pushed the utility angle there further yet, with just the Groovy Wrapper and the new utility scripts prepared recently - more coming, hopefully
    • Agree Agree x 1
  6. oniatus

    oniatus Member

    My main reason for a single repo is the developer workflow to keep the engine up to date.
    From the perspective of a developer who checks out the engine repo(s) once, then after 3 months wants to continue and be up to date:
    In the current state it would be a simple pull on the develop branch. If we split it in different repos, one would have to pull and manage multiple repos.
    This is something I find difficult on the modules from time to time, e.g. keeping track of Pathfinding, Behaviors, the engine and 2-3 other related modules for the behavior changes was a pain :)
    Should we do so, there must be an easy workflow for a developer to keep up to date, otherwise we may introduce viscosity in the environment which will also lead to bad code and workarounds :)

    For CI: I think it is okay to build all projects in the engine repo on each change and maybe improve later on if it seems reasonable.
    We are building the entire engine codebase on each PR right now and the split will not increase the amount of code to build, only the amount of projects -> and project resolution should be pretty fast with gradle, compiling, tests and static analysis should take most of the build time, right?
    So I think we would not lose something this way. Instead we gain some places for later optimization on the CI side, if we can build artifacts independ from each other :thumbsup:
  7. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    Absolutely agreed on the utility angle - more is needed to sustain split-repos. And it is doable too, the delicious luxury of "groovyw module recurse JoshariaSurvival" to then get every single JS module in source form in one go... yum :)

    A super build probably would be slower than current, but yes, I don't imagine it'll be much slower. One drawback is if a tool expects only one "thing" - for instance at present the Jenkins javadoc publish expects only a single path, so it can't grab javadoc from independent sub projects and publish all of them. But in that case we could simply rely on that super shiny new doc site somebody cool made for us :D

    We'd have to dig a bit deeper to see if there are any similar trouble spots, and how much hassle it would be to either get around them or live with a poorer result for something.

    Thoughts on the "engine2" bit and API layering? If that even makes sense as a term.
  8. oniatus

    oniatus Member

    "engine2" as API layer sounds good, we may even call it engine-api then.
    A separate project will also force us to touch everything we need at least once :)
    • Like Like x 1
  9. Cervator

    Cervator Project Lead and Community Wizard Staff Member

    Tiny side note while I'm getting exposed to some things of interest at GitHub Universe: the new "code owners" feature might work well with these kinds of sub components if they remain in a single repo. Essentially you have a config file in the repo that declares specific dirs to be "owned" by specific users or teams which enables some more review options. We somewhat discourage explicit ownership, but another way to think of it would be expertise or responsibility in making sure the area is taken care of.
    • Agree Agree x 1

Share This Page