Extensibility!

[Cervator]

So some related topics have come up here and there lately, but get jumbled together and could use some clarity

Extensibility is a heavy focus for us, that's no big surprise, that's a big part of what makes sandbox worlds so irresistible. User friendliness inside and outside the code base is paramount, and metouto is my classic example of how well we can invite in user contributions already

We don't have a lot of concrete modding implementation so far, and do need to get there. Some important questions are at hand for that.

1) Language. We could write everything in Java, but probably want to take advantage of a dynamic language for work outside the core, a.k.a. a "modding language". My initial personal preference was using Groovy as it is super easy and keeps the barrier of entry low. It also runs on the JVM and can do API-less access for development and debugging. Scala might have potential too. Yet Javascript could also be embedded via Rhino and ironchefpython has been advocating its advantages on higher activity than something like Groovy. Although I think JS might be a little less easy to work with?

As several people have noted we want to keep the total number of languages in check, although I'm still not sure whether to count Java/Groovy/Scala as more than one. JSON/GSON might also be replacing Groovy for "definition" files (data that I used to think might contain Groovy scripting to register with some sort of event handler / API)

2) API vs API-less - I think it would be easy to agree on wanting to make an API and forcing outright mods to use said API rather than allow direct calls into the codebase (currently doable via the GroovyConsole) or even outright replacing Java classes. Yet at the same time API-less debugging via the Groovy Console (or any equivalent console) is all kinds of, well, groovy. Maybe an advanced toggle available to users in a "Creative" mode. Multiplayer would eventually need a way to force no fancy console tricks, tho at the same time there's no need to limit it for a long time - cheating concerns isn't exactly a problem that's on the current horizon of interest.

3) Where does the core engine end and modding start? Internal modding / API usage could start very early, with many core systems implemented more or less as internal mods so they can be extended or replaced easily. Intending to convey the distinction here with when/where we want to enter "API-land" within the core game implementation as opposed to what is internal ES, in-game custom content, and out-of-game mods, tho it is a tricky distinction and I may be off my rocker here...

4) In-game modding tools and a somewhat related topic of where does the ES (game core) end and modding begin? This came up on IRC some, good discussion there. Second Life is a good example of what you can do, although you will never find a more wretched hive of scum and villainy nor more empty "rooms" so the game itself is not necessarily a model to emulate

Previously I had just figured we'd allow in-game (within the expectations of what the "game" is) block modification (imagine shrinking a 4x4x4 selection of block to a single block dynamically), but there might be potential in going beyond that, creating outright blocks, loading / modding shapes or textures, attaching scripts, etc. That's an advanced area and not one I'll step into without commitments from people willing to give it a shot, but it sounds like harrison (on IRC; soren renner on the forum) and ironchefpython might be up for something in that area

Also, I'm not sure how blurry the line would get between how flexible the ES and game itself would be, the possibility for user-generated content in-game (beyond the confines of the "game"), and outright external mods.


I know this post will come off as somewhat confusing and wall-of-text'y, so please do ask for clarification. I post it now so I won't take hours of waffling between descriptions that'll still feel lacking somewhere anyway

Finally, there's also some talk about knowledge exchange with the good folks over at Spout, who are all about this whole modding area (being a mod framework for Minecraft in a very brief and poor description, that's now evolving beyond that). We'd be foolish in not trying to learn what has worked and not worked in their experience, and likewise we've got client know-how they may feel free to inquire about

 

[ironchefpython]

1) Language

Here are the current language popularity rankings for 2012. http://www.tiobe.com/index.php/content/ ... index.html

I'll run down the list of dynamic scripting languages, and add a bit of commentary for each.

#6 PHP: Popular... but pony no.

#7 Visual Basic: A Basic variant might be a easily accessible language to write mods in, and Visual Basic is an established macro language... but there's no JVM support, plus... it's Basic.

#8 Python: I love Python, and it's the best language ever in the history of mankind. However Jython is meh. Still, if I was writing this strictly for my own use, I'd use Python, because I love it so deeply.

#9 Javascript: Everyone uses Javascript if they use the Internet, and almost all Javascript programming is event-driven code. Javascript is a first-class citizen on the JVM, with a mature and stable interpreter with Rhino, and there is active development on Nashorn. In my opinion, it's the perfect impedance match.

#10 Perl: If the objective is to make it easy for modders to read and learn from each other's code, Perl would hinder that severely.

#11 Ruby: Ruby is an interesting choice. It's powerful, flexible, and it's an up and coming language. Lots of people know it or want to learn it because Rails development is trendy. Jruby is an excellent implementation, and it's fairly stable.

#15 Lisp: Lisp... would be an *amazing* choice. Totally impractical, an all-around terrible choice, but utterly and amazingly cool.

#19 Logo: Not the first language you think of for an event-driven, asynchronous application scripting language. But on the plus side, we could make an actual in-game turtle that drew lines behind it.

#21 Lua: Lua is another interesting choice. There's been a lot of games that have used embedded lua for extension, so it's a pretty popular choice in the game modding niche. However, there is no "official" JVM implementation, of the 3 that have been made, there's not much activity around those projects.

#26 Fortran: Ah, the memories.

#28 Scheme: This would be a brilliant choice, except for the fact that nobody ever finishes a project written in Scheme. Programming in Scheme is all about the joy your brain gets about seeing an elegant solution to a tricky problem that
only a functional language like Scheme (or Lisp) would let you create. Programming in Scheme never seems to be about *finishing* a project once the interesting problems have all been solved.

#29 COBOL: I can't think of a single nice thing to say about COBOL with a straight face.

#30 Groovy: A perfect language for the task, assuming we'd be able to sandbox it so that Groovy scripts could *only* have access to the game engine through the API. The only downside is that it's less popular than COBOL.

 

[Immortius]

I see Java is #2 on that list. Given that groovy is a superset of Java, I can't see how the above popularity rating makes sense from a perspective of who would be willing to use it. I would prefer to see a selection based on more technical merits, albeit while avoiding the more outlandish options. I still think the Java API needs to be strong regardless, because this is an open source project and the Java API is always going to be exposed as a result.

In no particular order:

3) Where does the core engine end and modding start? Internal modding / API usage could start very early, with many core systems implemented more or less as internal mods so they can be extended or replaced easily. Intending to convey the distinction here with when/where we want to enter "API-land" within the core game implementation as opposed to what is internal ES, in-game custom content, and out-of-game mods, tho it is a tricky distinction and I may be off my rocker here...

I'm slowly moving in this direction. I'm thinking something along the lines of Engine (and thus, API defining) as

• The world/chunk/block handling + rendering
• Core rendering + camera
• The core of the GUI system
• Input handling
• The entity system and prefab system + persistence
• Backbone multiplayer when we get to it
• Asset types and managers
• Backbone infrastructure around game state, mod loading, and such
• Physics
• Config loading
• Interfaces and annotations used for extensions
• Scripting language support
• A small selection of component systems, components, and assets (i.e. Block, Location, Mesh components)
• Probably more stuff I've forgotten

Basically anything absolutely performance critical, anything that interacts directly with OpenGL/AL, anything not gameplay related.

Everything else should be extensions using defined extension points and interacting with the engine in a controlled manner.

So next there would be a Core mod which provides core gameplay, which most mods would want to work on top of

• Core entity events, components and component systems (characters, inventory, items)
• UI elements (screens, dialogs, HUD changes)
• Core assets (entity prefabs, textures, block shapes, block definitions, sounds, etc)
• Block systems? (for things like water and everything falling down?) Not sure.

Finally game mods (for specific game types) and just mod-mods.

4) In-game modding tools and a somewhat related topic of where does the ES (game core) end and modding begin? This came up on IRC some, good discussion there. Second Life is a good example of what you can do, although you will never find a more wretched hive of scum and villainy nor more empty "rooms" so the game itself is not necessarily a model to emulate

Well, an in-game tool for editing entities and entity prefabs would be an awesome start. In theory with a good set of components and underlying systems it would be possible to do quite a bit of modding even without scripting, just by mixing up the components and their fields that an entity has, and introducing new assets.

Also, I'm not sure how blurry the line would get between how flexible the ES and game itself would be, the possibility for user-generated content in-game (beyond the confines of the "game"), and outright external mods.

At the very least it would be nice to provide a friendly sandbox environment for developing mods and content, with features like dynamically reloading content, modifying entities in place, changing prefabs on the fly and so forth. Maybe even automatically detecting when content has changed? Would want all of this to be an option though, and disabled for normal play.

I imagine some gametypes may provide the ability to generate content as well, but within their own, balanced systems so not everyone can make the sword of kill everyone instantly.

JSON/GSON might also be replacing Groovy for "definition" files (data that I used to think might contain Groovy scripting to register with some sort of event handler / API)

I'm leaning in this direction, although haven't made any move to action it yet. There are some oddities to Groovy definition files, and having simple data files in a more consistent format will allow them to be edited by external tools as desirable.

 

[Cervator]

First, thanks for the comprehensive language listing and humorous commentary, ironchefpython!

I should mention it was an IRC session with him and harrison that finally got me to write this thing, including some of the elements.

I also realize that I don't think I've ever mentioned that we can limit the reach of the Groovy Console, since the commands get interpreted in our GroovyManager class and we can enforce what we want there by toggle buttons in the settings and what not. We also have a poor man's beginning of an API in there, almost, in the form of the convenience commands like giveBlock

Much appreciated feedback from Immortius as well, matches pretty nicely with the discussion on IRC

 

[ironchefpython]

I see Java is #2 on that list. Given that groovy is a superset of Java,

http://schneide.wordpress.com/2010/12/2 ... t-of-java/

Idiomatic Groovy is quite different than idiomatic Java, and just because you have 15 years experience writing enterprise Java, doesn't mean you're going to be able to read Groovy source code.

I can't see how the above popularity rating makes sense from a perspective of who would be willing to use it.

The reason nobody uses Groovy because it not-Java in a Java environment. There's a reason it's unpopular, it's because it brings dynamic dispatch to a world where developers expect static typing. Technically, Groovy is brilliant in its simplicity and power (even though it's slower on the JVM than both Javascript and JRuby), but Java developers don't use it because it's too not-Java, and non-Java developers don't use it because it's too much like Java.

I would prefer to see a selection based on more technical merits

Nope. If languages were chosen based on their technical merits, we'd all be writing in Lisp DSLs. Don't make the mistake of thinking just because something is perfect, that people will use it. We want to pick something popular, because of networking effects.

And we want to pick something portable to avoid lock-in. If Java graphics libraries get too slow or outdated, and the engine is re-written in C, there's Javascript. If we want to make an xbox version on top of .net, there's Javascript. And if we wanted to make a pure-browser WebGL/Voxel version, again, there's Javascript.

Groovy is perfect technically. But it's an evolutionary dead-end, there's no *energy* behind it. Groovy/Grails/Gradle even lost *inside* Spring Source, the entity responsible for Groovy development. (STS ships from Spring Source with Maven tools pre-installed, while Groovy tools are optional). If you moved the build system from Gradle to Ant/Ivy, you'd open up development to more people. If you migrated the Groovy definitions to JSON, your data could be processed by more tools and libraries. And a Javascript mod language would mean hundreds of times more potential mod developers.

Javascript is white-hot right now, there are a lot of developers building tools and libraries and fast implementations/JITs and books and best practices and web/app servers that run Javascript and databases that run Javascript, etc.

But that's just like my opinion, man.

 

[Cervator]

The reason nobody uses Groovy because it not-Java in a Java environment

Whut.. but it is Java :?

Same thing with Scala. You can take a perfectly normal Java class, change the extension, and compile it with Groovy/Scala. On top of that then you can replace a few awkward pieces of code with more smooth Groovy and that's all. I'd say that yeah, if you have 15 years of enterprise Java XP then you can totally read Groovy - it is no more unfamiliar than a library you haven't used before.

Sure, if you start going nuts with Closures and other fancy language features like that then it can feel a little different. It took me a few days to get used to those and the -> operator, but that was it. Now I look up Groovy reference examples just like I would Java reference examples.

I don't at all doubt that JS is hectic-popular right now, and maybe Groovy/Scala aren't huge, in part since they're likely often just attachments to Java projects. But they certainly aren't dead, even if they aren't apt for every purpose

That's not to say I'm closed to the potential of JS, or set on sticking with Groovy (and mayby Scala), not at all! I'm just not buying some of those arguments against it.

I did not know about the potential of using JS for xbox stuff and such, but then any option abandoning Java - honestly, we're doing a Java project due to being fans of Java, even if there might be lost potential in going with C. I don't see the Java libs going slow enough to doom the whole language, making us do a total rewrite. That I'm pretty sure we can etch in stone, otherwise we'd be talking a whole other project.

I think my main concern about JS is that if you're thinking Groovy is too different from Java, how different is Javascript? I think any 15-year veteran can pick up just about any language equally, but would find more differences trying to flow from Java into Javascript than into Groovy/Scala. And popular or not, isn't JS just more awkward than the niceties of G/S? Thinking newbie friendliness here, over width and popularity of the language (and while small solo, G/S both inherit the technical width from Java)

Edit: I enjoyed this quote from IRC enough to want to just leave it here (for programming language selection):

[10] <ironchefpython_> Elvis' bloated corpse has more life in him than Oberon.

 

[ironchefpython]

Whut.. but it is Java :?

No, it *isn't* Java. If I take C, and I add automatic memory management/garbage collection, and classes, and dynamic method dispatch, I might have something that looks like a superset of C, but it is fundamentally different. Not just because of the shitty performance, but because the idioms are all different. The semantics suddenly change, pure C might compile and run differently in my new language because of incompatibilities in type handling.

The behavior differences between Groovy and Java when running "pure" Java code are subtle, but they are there. To say nothing of how massively different the mindset is, you create anonymous inner classes for Java callbacks all the time, but you'd be utterly mad to do the same thing in Groovy.

I stand by my original assertion. The people who use Java do so because they like the things about Java that I hate about Java. An inflexible type system. The object-oriented straight-jacket. Massive verbosity. And the fact that Java does a better job of reducing developers to interchangable cogs than any other language in existence.

Groovy breaks all of those things, which is why no Java enterprise will touch Groovy. And if you're not already a Java developer, there's no reason to look twice at Groovy, despite the cleanliness and flexible design of the language.

I think my main concern about JS is that if you're thinking Groovy is too different from Java, how different is Javascript?

Javascript is completely different from Java. And this is a strength, and not a weakness. Just like browsers are written in C, and run "apps" written in Javascript, or like databases are written in C (or whatever), and database "apps" are written in SQL, and WoW is written in C, and WoW mods are written in Lua....

Having a strong separation between the modding environment and the game environment actually contributes to a cleaner design, and more mods that are more compatible with each other. Minecraft is a perfect example of modding gone wrong... all the mods are in Java, and none of them work with each other.

 

[Cervator]

I think we're getting somewhere good here, especially with all the discussion going on in parallel on IRC

Mainly I want to highlight the difference between core and modding, since they're different perspectives. I don't really want modders to have straight access to game state via Groovy, and am not at all even sure you'd want mods in Groovy or Java

And conversely ironchefpython isn't concerned with exactly how the engine is built (language-wise) behind the API wall, at least not in the context of modding. If at some point there's justification to mix in Groovy or Scala in there, that would be independent of the modding context.

We might have to agree to disagree about the "feel" to languages like Groovy or Scala, but I will admit inexperience and open-mindedness to eventually change my position as I learn more about them

Edit: Can't really let this IRC snippet go either:

[11] <ironchefpython_> Hell, the engine could be built on top of Doom, and as long as it's fun to play and write mods for, I don't give a damn if Keebler elves bake every single pixel with love and hand-dip it in chocolate.

 

[woodspeople]

A few cents of input, if you like. In extending a system there is a balance between power for the knowledgeable and safeguards for the blundering. The more you make possible the more you make breakable. Systems that "box people in" to a nice safe place keep them from breaking things but also keep them from improving things. I have been on both sides of this balance. I've built things that pro users loved but that novices couldn't handle; and I've made things novices loved but that pro users found too simple and limiting to be bothered with. Though to be honest, I've made the mistake of favoring pro users more often than novices, and I'm starting to think that may be the bigger mistake.

The beauty of open source is that the knowledgeable can just join in and deal with the real code, not a deobfuscated mess. So to some extent you are more free with open source to push the power/safety ratio down. Which means simpler ways to make changes with more protections against breaking things. Especially in a wide-open game it may be more important to favor cool ideas than technical sophistication. For example wouldn't it be amazing to get a real cultural anthropologist or biologist or historian or paleontologist or political scientist or urban planner or linguist or medical practitioner or scriptwriter to mod entity behaviors? Then, what is it worth to get that? And what would you need to do to get it? Maybe more API and less core? Maybe it is worth it to even think about jumping OVER the boundary all the way to a modding GUI. It is hard for programmers to believe this, but for many smart people out there even a simple text file is impossible to consider, when the text has to be structured. I don't know how many times I've sat with people who are perfect experts in some field and who can't get it that there has to be a semicolon at the start of a line, or a tab between two numbers, or whatever. Sometimes doing something as simple as giving them a form to fill out can unleash vast amounts of creativity.

As a test example, here's an idea we just came up with this morning for a mod: treaties. After a long and perilous search you find the head of some type of monster, say the queen of the skeletons or the elected leader of the pigs. If you provide the right gifts and have displayed the right behavior in the past (reputation?) he or she will listen to you and negotiate a treaty. Perhaps you will avoid mining any ore of a particular type, or you will give each skeleton you see something, or some other set of agreements will be put in place. Then if you ever break the rules all bets are off and you have to start all over. I can imagine manifold details to this. Like, on a multiplayer server who negotiates? What if one player signs a treaty and another breaks it? What if different player factions agree to different treaties? What if you kill a something-or-other in self-defense just before you find the king/queen/president/leader and you forfeit all possibility of negotiation for ten days? What if the monster who SAID they were the leader was an imposter and had no right to negotiate? What if individual monsters start breaking the treaty because the leader has been losing power? What if you lose your copy of the treaty and can no longer hold it out to show every something-or-other you meet? What if terms that seemed reasonable at first start to seem onerous? Should you break the treaty? Are you sure you know what will happen as a result? What if you break one treaty and those monsters tell the other monsters, and soon all your treaties are in shambles and everyone is attacking you?

My point is that the person who could write THAT mod and make it really fascinating may or may not be the person who could write the CODE that would make such a mod work. In fact in my experience they are rarely the same person. The big problem in computer gaming, IMHO, is that while the doors are wide open to technical expertise, they are mostly shut to imaginative-but-non-technical expertise. People should team up and cross over but they don't, much. That is why we get cool looking games with stupid boring simple behavior (it's a zombie, kill it) and boring repetitive challenges (kill lots of zombies) instead of real imagination.

Something to keep in mind! Hopefully helpful

 

[woodspeople]

Two more things and I'm done.

Groovy breaks all of those things, which is why no Java enterprise will touch Groovy. And if you're not already a Java developer, there's no reason to look twice at Groovy, despite the cleanliness and flexible design of the language.

My husband (who is currently employed writing Java) said JS is far better than Groovy for extensions and why are they using that. I forget why he said that, but I think it was something vaguely along the lines of what you said. Also for the lower technical expertise involved in modding flexibility can be a problem, not a solution. Flexibility is more of a barrier to novices than it is a boon to experts.

And:

Having a strong separation between the modding environment and the game environment actually contributes to a cleaner design, and more mods that are more compatible with each other. Minecraft is a perfect example of modding gone wrong... all the mods are in Java, and none of them work with each other.

Strongly agree with this statement. In fact the more different is the modding from the core the better things usually are. I think some of the problems with MC modding have come about because the MC developers can't seem to keep the line between core and modding in view, and keep obliterating it by mistake. They say something is open to modding, then they override it, or they say something can't be modded, then the modders mod it. That's probably poor coding practice and poor discipline as well, but I would guess using the same language might contribute to it.

 

[ironchefpython]

MASSIVE WALL OF TEXT

So you're suggesting... LaTeX?

Seriously though, when you have an open source project with an API that limits what you can do in that environment, you check it out from source control, extend the API, and submit your changes. I wouldn't see this project as being different. The modding API would be there to help constrain choice to improve productivity, when this became a hardship, it means it's time to augment the API.

 

[Cervator]

Massive walls of text best text!

I might need to dust off my feeble JS skills of lack thereof

 

[Cervator]

More spirited debate on IRC, featuring ROY SMASH with a tidbit of doubt near the end and more research needed

http://pastebin.com/Vn17qcXT

 

[Immortius]

I find that I generally agree with RobAwesome in his opinions - sounds like we've had some common experiences and come to some similar conclusions. Probably main difference is his modding background with Source Engine, while mine is more Unreal Engine. Certainly C++ with Mono for scripting is something I've considered.

woodspeople also makes some good points. I guess one question is what is the goal of the modding API(s) - is it the general way all gameplay is added onto the engine (like UnrealScript, or scripting in Unity)? Or is it a speciality tool for a specific purpose? A quick, and simple (but limited) way amateur programmers can add some behaviour is the one woodpeoples mentions. Something like AI behaviour is also comes to mind, as features like states (UnrealScript) or coroutines (Unity) can simplify designing these. Although I guess to get those features you need a DSL or perhaps a diagramatic tool.

My read on the API you've been describing, ironchefpython, is that it falls firmly in the latter camp since it relies on very high level methods from elsewhere? Would you agree with that? Which is fine, and I can agree with Javascript in that space, but we also need a solution in the former camp in my opinion. And I guess I'm suggesting Java for that.

Actually providing modding without code is also something I have been thinking a lot about. The entity system approach means that a lot of behaviour is data-driven, just a matter of the settings on an entity. And changing data dynamically is easy. Don't need to rely on fancy functionality or JITed scripting languages. If we had prefab loading integrated, just using the components we have right now we could:

• Make a bag that sits in your inventory that can be accessed by using it
• Make a block that explodes when used
• Create a tool that digs dirt faster

without writing a line of code. With some more work on the components a lot will be possible this way, and by integrating prefab editing tools into the game it could be done without messing with files. Actually supporting assets like meshes and textures properly can only help with this as well.

 

[Cervator]

Alrighty, following up here after reviewing assorted stuff around here for the milestone completion.

ironchefpython - extensibility is coming up and you've put good theory together and even pretty thorough-looking code / pseudo-code over in that GDoc (just checked it again). We had the big talk on IRC, although much focus was put on language musings that don't really relate much to implementation here since we have a for sure engine in Java. So my challenge for you then: Take the existing stuff and write up an integrated proof of concept running JS via Rhino inside Terasology and test how well it works / performs (akin to Immortius' POC on using OrientDB with the ES)

I'm not committing to use JS for sure, but the last big obstacle to something like that happening would be seeing it in action.

Special focus on modding the UI - as an example do an inventory slot mod like in the GDoc?

* Related thread viewtopic.php?f=4&t=247

* Related issue https://github.com/MovingBlocks/Terasology/issues/190 (or at least I'm curious exactly how it relates)

* Related potential: Maybe rapodaca has some good feedback / ideas / code to add there? (Previous discussion / work via that issue)

* Related system: Need to see this will integrate more / better with the ES

And the API itself and where what pieces go

* Thread: viewtopic.php?f=4&t=245

* In-game data editing - simply changing ES Component data and which component an entity possesses. Is this integral to the engine and on the inside of the API wall? In theory it would just edit whatever it can query via the ES. Example: Change move speed of a GelCube, attach a spawner component to a Gelcube. This could probably be done via GUIs (potentially temporarily via debug tools and clicking things)

* In-game console work - a debug tool for sure, but should some of it be exposed to powerusers if enabled? Can we interact with Java game state via JS, or should it remain Groovy? Or can it even be Java or Scala? Should an API be involved here?

* In-game advanced content / modding - a la Second Life's user-created persistent content. Ability to change model/texture stuff and attach scripts to objects. Example: Add a trigger to GelCubes that executes a brief script (like playing a small sfx) that executes randomly once in a while, maybe makes it visibly do something quirky. Would probably need GUI elements, would interact both via API and edit data. How would we save / export created content? Could you build a model out of blocks, then shrink it to a creature rather than a small detailed block, imbuing it with life? (viewtopic.php?p=1817#p1817)

* Out-of-game simple data-based extensibility. Like the original intent to just leave block definitions + textures in a directory and the game would load it automatically. Doesn't really interact with API or need a language, outside of a possible definition DSL the user would have to follow? Which would be extremely easy for blocks (make the definition name and texture match, include extra textures via specific properties, like now)

* Out-of-game actual mod, like adding a new slot complete with validation with what sort of object could go there (GDoc) and what it could do under what circumstance. Or maybe a new tool? Would need to follow API, but could it simply trigger things in the core so the engine doesn't need to call the external code after initially reading it? In other words, could it just cram everything related into the ES? How do we add anything to the ES externally?

* Out-of-game mod replacement of existing functionality deep in the code, registering (replacing) some sort of event that should be executed on a regular basis by the engine. Full API need and written in specific language. Performance implications by repeatedly executing stuff in a different language? Or can tech like JIT allow us to reach Java-level performance with JS?

 

[ironchefpython]

I find that I generally agree with RobAwesome in his opinions - sounds like we've had some common experiences and come to some similar conclusions. Probably main difference is his modding background with Source Engine, while mine is more Unreal Engine. Certainly C++ with Mono for scripting is something I've considered.

I don't disagree with Rob regarding the performance of C++ of graphical engines, where I diverge from Rob is in perceived performance of high-level garbage collected scripting languages in 2012. He's right that Javascript would be stupid in 2001, but wrong when he feels it is unacceptable slow for today's computers.

woodspeople also makes some good points. I guess one question is what is the goal of the modding API(s) - is it the general way all gameplay is added onto the engine (like UnrealScript, or scripting in Unity)? Or is it a speciality tool for a specific purpose? A quick, and simple (but limited) way amateur programmers can add some behaviour is the one woodpeoples mentions. Something like AI behaviour is also comes to mind, as features like states (UnrealScript) or coroutines (Unity) can simplify designing these. Although I guess to get those features you need a DSL or perhaps a diagramatic tool.

Given that JSON is as close to a "native" data format of Javascript as you can get, if you wanted to have a tool that generated JSON data structures, it would allow people to modify the data of mods without programming.

My read on the API you've been describing, ironchefpython, is that it falls firmly in the latter camp since it relies on very high level methods from elsewhere?

Nope. I would see the engine as generating events that are handled with javascript (like mousedown, keydown, collision, movement, etc.). The javascript handlers for these generic events could then create new events, such as block harvest, or npc damage, or block interaction. These events would be handled by specialized handlers that could spawn items, change npc behavior, or open doors.

Actually providing modding without code is also something I have been thinking a lot about. The entity system approach means that a lot of behaviour is data-driven, just a matter of the settings on an entity. And changing data dynamically is easy. Don't need to rely on fancy functionality or JITed scripting languages. If we had prefab loading integrated, just using the components we have right now we could:

• Make a bag that sits in your inventory that can be accessed by using it
• Make a block that explodes when used
• Create a tool that digs dirt faster

without writing a line of code. With some more work on the components a lot will be possible this way, and by integrating prefab editing tools into the game it could be done without messing with files. Actually supporting assets like meshes and textures properly can only help with this as well.

Again, I would suggest if a tool could generate JSON, this JSON would be interpreted as a mod, which would create new blocks or items with the behaviors enumerated in the JSON.

 

[ironchefpython]

So my challenge for you then: Take the existing stuff and write up an integrated proof of concept running JS via Rhino inside Terasology and test how well it works / performs

I'll see what I have time for this week.

 

[Cervator]

Cool, looking forward to it. In the meantime I'm curious what we can do to classify the different levels of functionality like hinted at above, any thoughts?

 

[ironchefpython]

Assuming that a "mod" is a collection of three types of things. Models, Textures, and Javascript/JSON. And mods are stored in a directory or ZIP file, either locally or remotely on an HTTP server.

* In-game data editing - simply changing ES Component data and which component an entity possesses. Is this integral to the engine and on the inside of the API wall? In theory it would just edit whatever it can query via the ES. Example: Change move speed of a GelCube, attach a spawner component to a Gelcube. This could probably be done via GUIs (potentially temporarily via debug tools and clicking things)

Editing textures would be done via a UI widget that displayed a bitmap and allowed editing. Editing (voxel based) models would be via creating a model view similar to the world view, and allowing addition and removal of voxels. Editing Javascript/JSON would use an in-game text editor.

* In-game console work - a debug tool for sure, but should some of it be exposed to powerusers if enabled? Can we interact with Java game state via JS, or should it remain Groovy? Or can it even be Java or Scala? Should an API be involved here?

The in-game console should be a Javascript REPL that can interact with loaded mods and the Engine mod api.

If you want a groovy console that can modify the Java state (with the ability to "break" the game), you can make one for debugging purposes.

* In-game advanced content / modding - a la Second Life's user-created persistent content. Ability to change model/texture stuff and attach scripts to objects. Example: Add a trigger to GelCubes that executes a brief script (like playing a small sfx) that executes randomly once in a while, maybe makes it visibly do something quirky. Would probably need GUI elements, would interact both via API and edit data. How would we save / export created content? Could you build a model out of blocks, then shrink it to a creature rather than a small detailed block, imbuing it with life? (viewtopic.php?p=1817#p1817)

A mod is model/texture/Javascript. Making GelCubes play a sound would be a javascript event handler attached to the GelCubes definition. Making a new creature would be making a model, and creating a JSON definition for that new creature that uses that model.

* Out-of-game simple data-based extensibility. Like the original intent to just leave block definitions + textures in a directory and the game would load it automatically. Doesn't really interact with API or need a language, outside of a possible definition DSL the user would have to follow? Which would be extremely easy for blocks (make the definition name and texture match, include extra textures via specific properties, like now)

Change the JSON to change the block definitions.

* Out-of-game actual mod, like adding a new slot complete with validation with what sort of object could go there (GDoc) and what it could do under what circumstance. Or maybe a new tool? Would need to follow API, but could it simply trigger things in the core so the engine doesn't need to call the external code after initially reading it? In other words, could it just cram everything related into the ES? How do we add anything to the ES externally?

Add textures if necessary to overlay new UI elements. Write javascript to handle mouseovers and item pickups and keybind events.

* Out-of-game mod replacement of existing functionality deep in the code, registering (replacing) some sort of event that should be executed on a regular basis by the engine. Full API need and written in specific language. Performance implications by repeatedly executing stuff in a different language? Or can tech like JIT allow us to reach Java-level performance with JS?

You would not be able to change the Engine implementation with a mod (i.e., you could not change rendering technologies, you could not modify the networking protocols, etc.). However, you would be able to tell the Engine not to load the vanilla mod, and instead run your own mod with your own block/crafting/harvesting/movement logic. If you wanted to, you could get rid of all the mining and crafting part of the game, and add guns and turn it into a TF2-style capture the flag game.

 

[Immortius]

Given that JSON is as close to a "native" data format of Javascript as you can get, if you wanted to have a tool that generated JSON data structures, it would allow people to modify the data of mods without programming.

It is a nice, human-friendly format regardless, so that is the direction I've been heading.

Nope. I would see the engine as generating events that are handled with javascript (like mousedown, keydown, collision, movement, etc.). The javascript handlers for these generic events could then create new events, such as block harvest, or npc damage, or block interaction. These events would be handled by specialized handlers that could spawn items, change npc behavior, or open doors.

Half those handlers are not engine functionality though - is the open doors handler also written in JavaScript? If not I don't feel this API is suitable as the primary means of extension. Event handling and propagation alone is not enough.

Again, I would suggest if a tool could generate JSON, this JSON would be interpreted as a mod, which would create new blocks or items with the behaviors enumerated in the JSON.

This is mostly done - entity prefabs are specified via json, and can be linked to a block definition.

In general I'm working towards a modding framework for assets + block definitions + prefabs + java at the moment. I'll post what I have in mind later.