Project Modding Arc

[Immortius]

Name: Modding Arc
Summary: Complete Modding support
Scope: Engine
Current Goal: Block Support Improvements
Phase: Implementation
Curator: Immortius
Related: Game Mode Config

The modding arc covers a range of improvements aimed at providing basic mod support without having to change the engine itself:

Block Support Improvements

• Clean up the block format, reviewing and consolidating the existing options
• Change blocks and block families to be identified by a uri that includes source mod
• Move block setup code out of groovy (maintainability)
• Remove the caching of the world tileset on disk, regenerate each time.
• Move block manifest into world manifest (fix the need to wipe save games on change)
• Move blocks to use a JSON format
• Change block URIs to incorporate shape
• Add block categories
• Implement lazy initialisation of blocks
• Add support for inlining prefabs (which can be used for extension information)
• Add block definition inheritance
• Move common information from Block to BlockFamily if sensible
• Move some blocks into mods
• Add HD block tile support?
• Documentation

General Modding

• Fix bootstrap process so that blocks and all other assets can be read from mod packages
• Add mod selection menu + configuration
• Add mod dependency specification to modinfo.txt, use during build/check at runtime
• Improve handling of type changes by entity persistence (auto conversion of similar types where possible e.g. int <-> float <-> double)
• Add some more types to be handled by entity persistence (e.g. sets)
• Improve error reporting on mod load - in game prompt/logging

Modding Code

• Add java code support to mods, specifically use of reflection to add component systems and the like
• Update gradle build script to create subprojects for mods, and to compile mod code
• Add sandboxing of mod code
• Some initial work on the mod api
• Move some code out of the engine and into mods.

Other thoughts:

• Could add support for a modified JSON format with some simplified syntax (commas optional, identifier not requiring quotes) and extra features (array size specification) - probably not worth it in the short term.

 

[Cervator]

Yay good stuff. I don't mind letting go of Groovy - does that include the Console tho and if so how do we replace it? Not that it is actually doing anything dynamic atm, just the potential is there

Also, and this might be another independent feature, I think we need a Block Browser of some sort in-game, probably both as a pre-world selector / edit thing and as a help function

 

[Immortius]

I don't really mind the console using groovy. It would be fairly easy to have something other than it if we are just using it for commands though.

 

[AlbireoX]

Miniions are written in Java. So it would make sense that should be a Java API if miniions were to truly be a mod. (They're in the mods package, so...)

I think we should put interfaces on everything (there are already anyway) and put them into a separate JAR. We can then combine the JARs somehow. It would make it clear what is logic and what is interface. This was one of the bigger problems I had when browsing Terasology's code.

Also, that means miniions could go in another JAR too when this is all done. It could eventually even be a plugin!

 

[Immortius]

I don't think we should have interfaces around *everything*. It would be silly to have interfaces around our math classes for instance (to the extent we can even do that).

I do like the idea of having an API jar that gathers together everything that should be accessible for mods. This would include interfaces and enums, but also include math classes (at least until they end up in a maths jar I guess), components, annotations, and any other common/utility classes. It would be a fair amount of work to make the separation though, and has some ramifications on things like the UI system.

However having an explicit API JAR isn't a requirement to allow mod/plugin jars - they can just work against the main JAR to start with. If/when the API jar becomes available the dependency can be seamlessly switched though. So I would get that working first. Having miniions as a separate plugin/mod jar would be part of that initial effort.

Also you don't really need to combine JARs as such, just have them as dependencies. The main Terasology.jar would depend on the API jar, and mods would depend on the API jar.

 

[dei]

I'd suggest if we get rid of groovy for block definitions anyway, lets write them directly as singleton-Java-Classes (or descendants of them). So we would have only one Language for modding, syntax-check for free (java compiler) including code completion and no parser needed. And a simple java class with some final properties isn't much more complicated than a json/groovy object to write for a modder (he will be using a template anyway).

At work we use our own framework wich uses 100% pure java-classes even for predefined DB-Objects and sql-queries and it makes coding reeally error-proof. You already see an error as a developer in the IDE or at latest when compiling without the need to test all queries and system-DB-Entries etc. Compiling works great using our own superpackage infrastructure. So if we already have Java let's use its features for the whole game.

Someone said he has something with API-Marker-Interfaces in the works, thought it was you Immortius, or was it begla...

 

[Cervator]

I'd personally disagree with going to Java classes for definitions, tho I don't mind changing them from Groovy to JSON or what not. Sure, your typical modder could probably write a Java class instead, but it is more work. You can literally add a new block to the game right now by creating an empty (or one line?) file (I'm even thinking about definition-less blocks that trigger off a block texture simply being present). Anybody can do that, with Notepad and Paint.

There are a whole lot more anybodies out there than there are modders - and some of those anybodies will likely become modders thanks to the lowered barrier to entry

On top of that we'll be moving toward letting users create custom content inside the game that should write out the custom objects as super simple human-readable config files (definitions). Both blocks, shapes, etc.

 

[Immortius]

There are three reasons why I would tend towards using straight data files over code files where possible:

1. Tooling. Much easier for tools to be written to work with JSON than to work with Java files. If down the track someone wants to create an external block editor or viewer, in any language, then by keeping to simple data files we enable it.
2. Security. Data files are somewhat safer, as long as care is taken around buffer overflows and input validation. Code files can do all sorts of things. Sure, we could sandbox them and restrict what they can do, but it is simpler just to avoid that pit-trap entirely.
3. Complexity. The full power of is programming language is overkill for the problem space. Having to compile them (even if we can take care of that in the engine) just adds to this.

I've been thinking it would be nice to have a simple block format for adding a set of blocks at once around a single texture - so both the full block, and other shapes like stairs, half-blocks, slopes, etc as desired.

 

[dei]

Totally risking being known as the java priest from now on :

Cervator :
"add a new block to the game right now by creating an empty (or one line?) file"
In what would that be changed when writing it in Java? It would look as simple as this: public class SomeNewBlock extends SimpleWorldBlock {}

Immortius :
1. OK will be a little bit simpler, if the tool isn't written in Java anyway
2. We'll write a manifest for java's security manager, and there's no difference in security (you can mess around in source- and bytecode anyway)
3. You could write different levels of superclasses like "SimpleWorldBlock" above which implements all nasty Java-methods but allows to overwrite them if wanted.

You're writing of avoiding pit traps: We're creating even more when writing an maintaining our own parser. We would limit functionality and would have to touch the parser every time someone wants the blocks to have a slightly new behaviour/property etc. You get parser-errors eventually even at play-time...

Blocks of the same type with different shapes, no problem --> simply add some marker-interfaces --> one file to maintain for all shapes

 

[Immortius]

Immortius :
1. OK will be a little bit simpler, if the tool isn't written in Java anyway

A java tool isn't going to have any advantage in dealing with Java classes. You're expecting a tool to be able to cope with the full complexity of a turing complete programming language and all its features in reading and editing a block?

2. We'll write a manifest for java's security manager, and there's no difference in security (you can mess around in source- and bytecode anyway)

Point still stands - that is making a bunch a work for no particular benefit. We will have to do this (plus custom class loaders) eventually for mod code though. But a major security principle is to limit attack vectors in the first place.

3. You could write different levels of superclasses like "SimpleWorldBlock" above which implements all nasty Java-methods but allows to overwrite them if wanted.

Different sort of complexity to what I meant. But worth pointing out that the syntax for writing a Java method is much clunkier that adding a JSON property (or even an xml element).

You're writing of avoiding pit traps: We're creating even more when writing an maintaining our own parser. We would limit functionality and would have to touch the parser every time someone wants the blocks to have a slightly new behaviour/property etc. You get parser-errors eventually even at play-time...

1. We're using GSON, not writing our own parser. In fact we can load the JSON directly into a class with GSON. Some processing of that data is required anyway as we are sometimes generating multiple blocks from a single definition.
2. I already mentioned having an extension interface for blocks - with JSON this is simple as we can just throw all the properties into a map.
3. On runtime checking... I don't see this as a big issue? If we are dynamically compiling provided java files, that is just runtime checking anyway. If we're expecting people to compile their block definitions that an annoyance and a pain, and we could equally provide a utility to test block definitions that fulfils the same role if it we wanted. We can also just have the game validate all the block definitions on start up, which is fundamentally the same.

And the thing is... a compile check is not sufficient to validate a java file! It could be full of NPEs, dividing by zero, accessing nulls, infinite loops. A data file can be validated because it is much more limited.

 

[Cervator]

Totally risking being known as the java priest from now on

Okay

Immortius answered everything pretty much. Main different I latch on to is - Groovy/JSON:

• Make empty file (or even just create a texture) in specific dir. Celebrate
• Run game, magic happens
• Optionally get fancier later and allow picking up new blocks with game running
• Create blocks in-game and have them written out to a definition file right there

Java:

• Make Java file, fill it with basic stuff that doesn't really matter (lose a few interested people already)
• Compile Java (lose lots more people that can't make that work - what's a classpath?)
• Run the game, magic happens
• Trouble adding blocks while running or easily saving new blocks?

The block structure already has a Java hierarchy that comes with defaults, methods etc., and the very location of the block definition file determines what it inherits. The block definitions can be crammed right into a database too. I admit that over time tweaking stuff will take some work, but I think that'll be the same in any case

 

[Immortius]

Although I'm tempted to throw away the java hierarchy and add a block template files that specify some values blocks can pick up:

// Template
{
    "title" : "plant",
    "requiresLight" : true,
    "allowsAttackment" : false
}
 
// Actual block
{
    "title" : "rose",
    "template" : "plant",
    // ... rose specific settings
}

Which allows users to add their own templates with defaults.

 

[Cervator]

I'd be fine with that. Main reasoning beyond defaults was being able to test for a specific block class for processing. But I figure you could do that with a property as well.

That, and Blocks started as pure classes in the beginning - my refactoring ability only goes so far

 

[Immortius]

I have begun work on this now. Main things that are complete at this time are:

• Review of block properties. Removed some unused properties, tweaked some of the names, split some properties.
• Block and Block Families are now identified a Block URI, similar to the Asset URI. A Block Family has the form <mod>:<family> (e.g. engine:stone, core:chest ), while a Block has the form <mod>:<family>[.<identifier>] (e.g. engine:stone, core:chest.left). For blocks belonging to families with a single block, the identifier is left off.
• Some work towards the JSON format, not yet complete.

For review, the current format is:
(everything is optional)

{
    // If not specified, is the ProperCase of the file name
    "displayName" : "Block",
    "liquid" : false,
 
    // Copies and then overrides settings from another block
    "basedOn" : "engine:plant",
    // If true, this block exists only for being based on, and shouldn't become an actual block
    "template" : false,
 
    // The shape of the block, cube if there are any issues
    "shape" : "engine:cube",
    // If specified, creates a block family based on the settings for each shape - not compatible with "rotation"
    "shapes" : ["engine:cube", "engine:stair", "engine:slope"]
 
    // The rotation type of the block. If horizontal or align to surface, then makes a block family combining the different rotations needed
    "rotation" : "none/horizontal/alignToSurface",
 
    "sides/top/bottom" : {
        // both which parts to have for align to surface, and what override settings they should have. Can override any of the other setting (allows torch to have a different shape when on the ground, for instance)
    },
 
 
    // Hardness 0 for indestructable, maybe make that a separate flag too?
    "hardness" : 3,
 
    // Can other blocks be attached to this. Should add settings for this per side of the block
    "attachmentAllowed" : true,
    // Whether this block should be destroyed if the below block is destroyed, should adapt this for
    // other cases like torches on walls.
    "supportRequired" : false,
 
    // Can the block be passed through like a liquid or air
    "penetrable" : false,
    // Can the block be targeted by the player
    "targetable" : true,
 
    "invisible" : false,
    // Whether the texture is masked (need a separate flag for semi-transparent?)
    "translucent" : false,
 
    // Should both sides of the mesh be rendered (for billboards)
    "doubleSided" : false,
    // Is Faux SSAO (shadows) created around this block
    "shadowCasting" : true,
    "waving" : false,
 
  "luminance" : 0,
 
    // The image to use.  Defaults to the tile with the same uri as the block.
    "tile" : "engine:stone",
    // Or can specify per side
    "tiles" : {
        "all" : "engine:stone",
        "sides" : "engine:stone",
        "topBottom" : "engine:stone",
        "center" : "engine:stone",
        "left" : "engine:stone",
        "right" : "engine:stone",
        "top" : "engine:stone",
        "bottom" : "engine:stone",
        "front" : "engine:stone",
        "back" : "engine:stone"
    },
 
    // Special color maps
    "colorSource" : "default/color_lut/foliage_lut",
    "colorSources" : {
        "all" : "default",
        // per side options again, same as tiles
    },
 
    // Color tinting, can be a 3 or 4 dimensional color.
    "colorOffset" : [1,1,1,1],
    "colorOffsets" : {
        // again, per side options as per tiles
    },
 
    // Not yet functional because of physics system work needed, but the mass of the block debris
    "mass" : 10,
 
    // Whether the block becomes debris
  "debrisOnDestroy" : true,
 
    "entity" : {
        // The entity prefab associated with this block (maybe allow it to be inlined?)
        "prefab" : "engine:chest",
        // Whether the entity should be persistant
        "temporary" : false,
    },
 
  "inventory" : {
        // Should the block go straight into the havester's inventory
        "directPickup" : false,
        "stackable" : true
    }
 
}

 

[Cervator]

Good stuff

I had an idea earlier that essentially you've got here with "shapes" - but am wondering if it could be taken one step further. Generating a Block class and ID for every material + shape will bloat at a fantastic rate, especially until we get to a blockID based on a short. Is it possible that we could initialize shapes beyond "Cube" and those hard set in actual definitions (torch, etc) at the point in time they're needed?

I added a few shapes for Marble just yesterday and it was easy enough but excessive. You could easily do that for dozens of other existing materials. Could we just catch the point in time when the game provokes the need for a shape, by crafting a specific block, running a giveBlock command in the console, world gen creates a slope of a specific material, etc? Or too annoying trying to catch that and add restrictions if ever needed? Like no thin poles made out of sand

 

[Immortius]

We don't create a block for every material + shape, only the combinations requested through block definitions. Or... are you suggesting ideas around having it simpler to put together those combinations without writing a whole bunch of blocks unnecessarily? Slightly confused, sorry.

Assuming the latter, what I had in mind was two things. I believe we need at least one block definition per material - we can't just make them from the images because we'ld get blocks like ChestTop. So then the goal is minimising the number of definitions you need to write, and then minimising how much needs to go in them.

So first thing is to support creating multiple shapes of a block from one definition

dirt.json
{
    "shapes" : ["engine:cube", "engine:slope", "engine:stair"]
}

This could be set up to automatically create a BlockFamily for each shape ("engine:dirtCube", "engine:dirtSlope", "engine:dirtStair"), with an appropriate type of family based on whether the shape is symmetric or not (Cube is, Slope and Stair are not so have 4 rotations each). Generating a total of 9 blocks.

Secondly is to have the "basedOn" system working, so you can have a bunch of reasonable templates to base a block on - BuildingMaterial.json for things which can be stairs, slopes, half-blocks, etc, EarthMaterial.json for things which have less.

Actually adding blocks at runtime as they are requested is possible. Probably simplest way would still be to load them all up and partially registering them (so they can be found by uri), but delaying the assigning of an id until actually used. I'll add that to the todo list. I could also just switch from byte to short - it is a pretty easy change at this point. Then the next limit would be the 256 unique block tile limit (can easily be expanded to 4096 though be changing the texture atlas to a 1024x1024 texture, maybe 16384 if we switch to 2048x2048 - would need to work out how well supported that is video-card wise).

 

[Cervator]

Actually adding blocks at runtime as they are requested is possible. Probably simplest way would still be to load them all up and partially registering them (so they can be found by uri), but delaying the assigning of an id until actually used.

Yep, that was pretty much what I meant. Doing a single block definition by material, and if "shape" or "shapes" isn't defined allow all shapes to be requested on demand and assigned a block ID then. Right now to build in Marble I had to add a few definitions - "shapes" would fix that to add a property to the single material, but then when I want to build in Corundum I'd have to go add shapes there. Unless it is already added, but then maybe if a new shape is added I'd have to add that somewhere, etc...

Pure texture-based blocks could potentially be done by just having a magic directory under /mods for "shortcut block textures" - complex blocks (multiple textures) would still need a definition.

 

[AlbireoX]

2048x2048 - would need to work out how well supported that is video-card wise).

I believe that is the limit, 2048x2048

 

[Immortius]

On some systems you can do ridiculous things - I did some tests with 16x16384 textures in the past. Makes it much easier to index tiles

On the blocks... both the auto-blocks from textures and auto-all-shapes when not specified are possible. There are two side effects - one is the uri for block families will now be [block-mod]:[block-def-name]:[shape-mod]:[shape-name] (e.g. engine:dirt:engine:slope ), with the latter two skipped for the default cube.

The other is potentially running out of ids at runtime and getting the air block instead of the request block.

 

[Immortius]

I'm moved the first wave of changes into develop:

• ImageManifest is no longer saved and loaded to disk, BlockManifest is now incorporated into a saved games's WorldManifest. This breaks existing saved games, but from now on saved games shouldn't break with the introduction (or removal) of blocks.
• Blocks now use the json format.
• Blocks can now be introduced through mods, without requiring a recompile.
• Support for quickly making blocks with just the image has been added - anywhere in a mod's blocktiles folder you can create an "auto" folder, and any tiles in that become blocks (only with the cube shape at the moment)
• Block definition inheritance has been added.
• All blocks are identified by uri now (e.g. "engine:dirt", "minerals:clay")
• Moved a lot of blocks out into mods.