Internationalisation

[Immortius]

I have been mulling how to do internationalisation for a while - it is somewhat tricky. Perhaps we need a special asset string type, "IntString" - these are identified by a uri identifying a module, localisation file and localisation item. e.g.

Some prefab:

{
    "DisplayName" {
        "name" : "core:items.axe"
    }
}

Localisation file (items.local in core/assets/localisation/en or something like that):

{
    ...
    "axe" : "Axe",
    ...
}

But the question that leaves is how to get this acting nicely with normal strings.

 

[Skaldarnar]

Possibly worth its own discussion thread. How about not using name for the key but something like "localeKey" or "i18n". This way you could either specify a string as name or point to the localisation file(?).

 

[Immortius]

It is only "Name" because that is the field of the DisplayNameComponent, if that is what you mean? That bit cannot change because a component can have multiple internationalisable fields. Probably could use a special character at the start of the value to indicate whether it is an internalisation reference vs an ordinary string:

{
    "DisplayName" : {
        "Name" : "Axe",
        "Description" : "#core:item.axeDesc"
    }
}

 

[Cervator]

Bump - wanted to post http://weblate.org in a translation-related spot. Came across it and it looks very spiffy and heavily GitHub integrated.

mkalb - maybe something to compare with the current setup for the Launcher, considering if we could extend a tool like Weblate across all the projects ?

Big question is how you need to mark / replace strings as it needs to be compatible with prefabs somehow

 

[Halamix2]

Check this: https://crowdin.net/page/cli-tool

 

[mkalb]

Thank you for the links. I will look at it soon. I want to divide the translation from the code releases.

 

[mkalb]

I added an Issue with some links.
https://github.com/MovingBlocks/TerasologyLauncher/issues/229

 

[Halamix2]

Little table, version 2.4 ( on Google docs):
https://docs.google.com/spreadsheets/d/1YWmw24_vx0YW4if0wczg2WmSt1DbVV_UAfRU81Ix_-E/edit?usp=sharing

 

[Cervator]

Added an entry for Pootle and found a couple links:

http://blog.cihar.com/archives/2012/05/11/pootle-vs-weblate/
https://www.openhub.net/p/compare?project_0=Weblate&project_1=Pootle

 

[Cervator]

http://translate.terasology.org is LIVE and tested!

It is running Weblate 1.9.2 via the bitnami installer, which is the newest version available via bitnami. There is a 2.0 available, which was released all of 4 days ago. Might be a while before that's available via bitnami, but I tried installing it manually first, and lets just say it went poorly ... It involved lots of manual compilation of dependencies, which eventually broke on a couple components and required me to bump up the droplet running it from the tiniest size (512mb memory) to the next step up (1gb). Via bitnami it seems to work fine on the tiny size droplet which is only $5/month

Much fun was had while setting this up over 2-3 days where I probably should be doing release work, but I made progress there too, so hey

I've got both GitHub OAuth working (go to login and click "github") as well as @Gooey pushing commits to GitHub, to the weblate branch for MovingBlocks/TerasologyLauncher. The commits get "authored" by the Weblate user saving the change (via their email matching GitHub user) then submitted by GooeyHub . I've also enabled the Weblate hook on GitHub which is supposed to make the app auto-pull latest when things get pushed to GitHub. Haven't tested that yet but it seems very basic and should be on by default.

Stuff still to do:

• Sign up some interested users and make them admins/managers. I shared the base admin user/pass with @Skaldarnar, @Halamix2, @mkalb, and @msteiger so you guys can log in and make yourselves admins after OAuthing separately with GitHub or what not.
  
• Make the involved services auto-start on a system reboot. Haven't looked at that and am tired
• Review the Performance Report (really should be "Health Report") under the admin panel to see if we need to fix some of the issues reported there.
  • Big warning: I tried to "fix" the domain issue by setting translate.terasology.org instead of example.com and irreparably broke the install Am loving droplets + bitnami, just reset to base image and reinstalled in 15 minutes, hah. I took a snapshot of the working system now but if we tinker more we should probably do so after another backup/snapshot
  • There is a bunch more instructions at Minetest's doc page for Weblate setup (thanks @Calinou for the pointer!) that might help fix additional things
• See if there's a problem with encoding somewhere. My test commit only intended to change a single . to ! for testing, yet resulted in a bunch of quirky edits. Maybe they do not matter? Dunno. Haven't tested the launcher
• Add another sub-project to the Launcher project. @Skaldarnar had set up two, I just recreated one.
• Decide how we want to use it. I figure keep Weblate pointed at its own branch, then ourselves make a PR or push to develop? Although then it might not get up to date data to translate. Too many commits if pointed right at develop?
• Update the related documentation. I dumped my install notes in the main wiki but would like somebody to review and maybe make it a little prettier as we get more into actually setting up and using different sub projects. We also need to link to it from some places, right now it is a poor little orphan page with no links.
  
• Move onwards to internationalizing Terasology itself!

I tested a Pootle install via bitnami on a droplet too, but it seems vastly more involved to set up with Git right. After a couple mistakes with Weblate config there seems a breeze

 

[Skaldarnar]

Great news it's working!

I've added the second sub-project real quick covering the `MessageLabels` bundle.

Regarding the further usage: I think the separate weblate branch is very useful for testing until everything is set-up 100%. Afterwards, all reviews should be accomplished within Weblate. I don't want to use the auto-push on commit feature of Weblate because that might clutter the commits. I'm not sure if users (normal user group) can initiate a push as well? If not, having the admins/managers review the translations and initiate a push to the main develop branch would work as well.

 

[Cervator]

The git maintenance options in Weblate can be access controlled, yep. I agree on no auto-pushing. You can make several saves then do one commit then push, or do a few more commits etc

 

[Halamix2]

• "Big warning" thing is fixed now,
• added CrashReporter (weblate branch).
• Weblate changes \! and \: to ! and : but Launcher still compile without errors (or at least it should )

 

[Immortius]

Details after today's discussion:

Considerations

• An international string must hold its "id", so that it can be persisted and transmitted over the network appearing correctly on remote machines with different language selected.
• Translations need to use files that work with existing translation tools such as weblate - one file per language containing an id->value map effectively.
• Translations need to work for strings in various assets (prefabs, nui) as well as strings in code.
• Changing language at runtime is not required to be supported, though it would be nice.

Implementation Thoughts

• We will need a I18nString type for internationalisation supporting strings. I would suggest this be an asset type, so the urn is the string's identifier. This supports the need for persisting the id. They can store a mapping of lang->value, and by default will use the current language of the java environment for toString();
  
• Alternatively, we do the translation from from the id to the actual string at the point of display. So we can continue to use strings as is, and if they start with an indicator character (e.g. "$core:item#axe") we translate them at that point. Would mean a lot more lookup calls though. Does give some flexibility interchanging international strings and normal strings.
  
• Translations files would be assets. The individual entries within a translation file would be identified by the urns structured like "module:translationFileId#entryId". e.g.

file itemNames-en.properties in core module

axe=Axe
sword=Sword

urns:
"core:itemNames#axe", "core:itemNames#sword"

• The default translation file would form the basis of the asset. The additional translation files would supplement this with additional language mappings. New modules can add to this using the asset delta system.

e.g.

core\assets\i18n\itemNames.properties <--- Main file
core\assets\i18n\itemNames-jp.properties <--- Language supplement
core\assets\i18n\itemNames-en.properties <--- Language supplement
frenchPack\deltas\core\i18n\itemNames-fr.properies <--- Language supplement

 

[Cervator]

So perhaps to keep down translation noise in the engine project we should make a single EngineLanguagePack module that uses deltas to translate the engine. Then we can point Weblate to that one module and include it as part of Iota.

Even if you exclude the module (we might do that for the applet) the game runs fine.

Other modules could contain their own i18n files, use a similar approach if they get big and noisy, or if for some reason a third party wishes to add a translation module with deltas outside of our usual infrastructure they could do so too.

Or maybe even Iota should use a single i18n module to centralize translations and noise across a large set of modules.

Although I wonder where in-game you would pick the language, as it would sort of make a hard module dependency active even in the main menu. Maybe we need a Translation module type that get loaded on game startup and cannot be disabled unless you revert to the default language or one in a different module.

If the engine or a module changes and makes new strings available some process needs to make that visible to language modules, so that in turn can become visible to Weblate. You could probably do that with some scanning in Jenkins and push forward updates with empty strings to the appropriate modules. Would take some automagic.

Then there's also the point to consider of having the main file in one repo yet language files in another.

Hmm. Yeah this is a bit trickier for the main game

 

[Cervator]

Edit: Mixed up my terms, new version supports JSON, old version had us using prop files

Okay, shiny new test server running Weblate 2.0 is now online: http://weblate2.terasology.org/

That should support the properties JSON files we discussed in a recent meeting. I promised to get this version up to test around with it some - ping @msteiger @Halamix2 @Skaldarnar !

It isn't hooked in to GitHub OAuth yet since it is just for testing, but I've authorized it with an SSH key on @Gooey's GitHub account so it should in theory be able to push to stuff. I suggest testing in a fork somewhere, maybe use a personal Launcher fork or tinker with the Nanoware one.

I'll send the admin user/pass to the three of you. Get me an SSH key sometime if you want root access (I think @Skaldarnar has one in there already). Same goes for anybody else interested in testing!

The droplet powering it is pretty weak, but earlier we hit errors even after upgrading to a more powerful droplet. Notifications from the app should go to gooey@terasology.org which forwards to me and @msteiger (translate@terasology.org stays only active on the live site). No errors yet!

 

[Halamix2]

Weblate 2 is ready for testing, with one (old) repo (TerasologyLauncher fork).

 

[Cervator]

Thanks @Halamix2

I double checked that the key in use should work with Gooey on GitHub, so we should be able to test in any needed repo.

Any of you need anything remaining from me? Will be happy to turn everything over and focus on something else

(Also: Fixed wrong term above, the new goal is JSON files, not prop files)

 

[msteiger]

I'm on it! There's a new branch "weblate" now:
https://github.com/MovingBlocks/Terasology/tree/weblate/engine/src/main/resources/assets/i18n

The translation files can be edited through Weblate:
http://translate.terasology.org/projects/terasology/

The next step will be about integrating the data in form of assets so it can be used (through I18nMap probably)

 

[Ruffian]

Resource not available - http://translate.terasology.org/