Custom word files in rule packs

Started by Elevator, August 30, 2017, 06:28:41 PM

Previous topic - Next topic

Elevator

A lot of languages have different forms of the same word: plural/singular, gender-specific, etc.
RulePackDefs in current implementation only allow to use one single form for each word, which is often not appropriate for different contexts.
For example, in English version NamerFactionPirate rule pack contains lines:

        <li>name->The [badassadjective] [groupname]</li>
        <li>name->The [badassadjective] [badassanimal]s</li>
        <li>name->The [badassadjective] [badassperson]s</li>

These rules generate faction names in plural form just by adding "s" to the end. This trick is only actual for English, but not for many other languages. Moreover, some languages, for example Russian, require using plural form of [badassadjective] (because the adjective is connected to the plural noun). This is really annoying thing that leads to poor quality of translation.

There is a way to get rid of this problem by adding extra <rulesRaw> in Def files:
  <RulePackDef>
    <defName>NamerFactionPirate</defName>
    <rulePack>
      <rulesStrings>
      ...
      </rulesStrings>
      <li Class="Rule_File">
        <keyword>badassanimal</keyword>
        <path>Words/Nouns/Animals_Badass</path>
      </li>
      <li Class="Rule_File">
        <keyword>badassadjective</keyword>
        <path>Words/Adjectives/Badass</path>
      </li>
      <li Class="Rule_File">
        <keyword>badassperson</keyword>
        <path>Words/Nouns/People_Badass</path>
      </li>
      <li Class="Rule_File">
        <keyword>badassconcept</keyword>
        <path>Words/Nouns/Concepts_Badass</path>
      </li>
      <li Class="Rule_File">
        <keyword>groupname</keyword>
        <path>Words/Nouns/GroupNames</path>
      </li>
      <li Class="Rule_File">
        <keyword>badasscolor</keyword>
        <path>Words/Nouns/Colors_Badass</path>
      </li>
      <!--EXTRA FILES-->
      <li Class="Rule_File">
        <keyword>extra</keyword>
        <path>Some_Empty_File</path>
      </li>
      <li Class="Rule_File">
        <keyword>extra</keyword>
        <path>Some_Empty_File</path>
      </li>
      <li Class="Rule_File">
        <keyword>extra</keyword>
        <path>Some_Empty_File</path>
      </li>
      <li Class="Rule_File">
        <keyword>extra</keyword>
        <path>Some_Empty_File</path>
      </li>
      <li Class="Rule_File">
        <keyword>extra</keyword>
        <path>Some_Empty_File</path>
      </li>
      <li Class="Rule_File">
        <keyword>extra</keyword>
        <path>Some_Empty_File</path>
      </li>
    </rulesRaw>
  </rulePack>
</RulePackDef>


This allows translators to add custom files with custom keywords, which could be used in rulesStrings:

<NamerFactionPirate.rulePack.rulesRaw.7.keyword>badassadjective_plural</NamerFactionPirate.rulePack.rulesRaw.7.keyword>
<NamerFactionPirate.rulePack.rulesRaw.7.path>Words/Adjectives/Badass_plural</NamerFactionPirate.rulePack.rulesRaw.7.path>
<NamerFactionPirate.rulePack.rulesRaw.8.keyword>badassperson_plural</NamerFactionPirate.rulePack.rulesRaw.8.keyword>
<NamerFactionPirate.rulePack.rulesRaw.8.path>Words/Nouns/People_Badass_plural</NamerFactionPirate.rulePack.rulesRaw.8.path>


Please add extra rulesRaws to RulePackDef files! We really want to make translation better and easier for players to read!

Harkeidos


Elevator

Sorry for the replying to the old topic. I just do not want to create another similar topic here.
So what about adding more keywords and file rules to be overriden in DefInjected? This is really important stuff for translation of story and name generators. This question concerns me as a translator very much.
It would be great if any deleloper posted their comment on it.

Tynan

I'm really sorry the translation systems haven't gotten more love.

Piotr (ison) has added some useful features that are in 1.0 unstable build. For example, variable size RulePacks so you don't have to translate line by line. He's going to modify the translation repos with that tomorrow.

For all the plural/gender/adjective-noun agreement/irregular counting stuff and more, I want to do it but it's hard to find the time!
Tynan Sylvester - @TynanSylvester - Tynan's Blog

Elevator

#4
Thank you for your reply.
I've just noticed the commit by Isonil. This full-list translation feature is very helpful for translation of generators.
The other important thing I appreciate in the new version is the reworked translation report generator. That's really great feature, thank you and your team very much!

The third thing I've noticed confused me a lot. The report contained tons of errors looking like:
Translated untranslatable field keyword ...
I've decompiled the dll and noticed the [NoTranslate] attrubute for "keyword" field in class Verse.Grammar.Rule.
All the keywords are connected to the Rule_File's (with keyword and text file path) I've overriden in DefInjections for B18. That's exactly what I mentioned in the OP.
As I mentioned, unfortunalety it's almost impossible to use current file rules in their initial state to achieve good quality of generated stories. I relied so much on the feature of overriding keywords and paths to make them work better. I spent quiet a lot of time trying to improve the quality of generated stories by making references to new text files (with plurals forms, differents genders and so on). I regret to imform that this new attribute just multiplied all this work by zero. Moth of the Russian generators are broken down now. We need your help!

I can see several ways to solve this situation.
That full-list injection feature looks quiet good. It will be perfect if you implement the same mechanism for File Rules. It will give us full freedom and flexibility! This is the best solution I could ever imagine.
Or al least, if it's possible, could you please just remove this [NoTranslate] attribute from keyword field? I don't think this operation will consume a lot of your time, but at least it will save the translation (generator programming) work that is already done. Moreover, if you copy-paste some (or a lot of) extra <rulesRaw> into the Mods\Core\Defs\RulePackDefs\RulePacks_Global.xml and other rulepacks you will save the tranlator's time and nerves in future as well :).

Thank you in advance.

Tynan

Sounds reasonable.

Let me get Piotr (ison) in here.

I wonder if it would make sense to have a [MayTranslate] attribute, to mark fields which don't create translation report issues when untranslated, but also don't throw errors when translated?
Tynan Sylvester - @TynanSylvester - Tynan's Blog

Elevator

#6
QuoteI wonder if it would make sense to have a [MayTranslate] attribute
That looks good. If you decide that the other approach (with full-list injection of file rules) is hard or impossible to implement, this [MayTranslate] solution will at least save the work done so far.

Anyway, if you want to discuss the full-list approach, feel free to contact me. As a developer, I'm able to help you with some technical issues if it's necessary.

ison

Alright, thanks for the feedback on translating rule packs. I realize they are one of the most difficult things to translate, so it's good to hear some input.

I think the best solution here would be to add a new field to rule packs: List<string> rulesFiles; and allow full-list translation.

This way it would be possible to add more files like this:


<CraftedArt.rulesFiles>
<li>newcustomtag->Words/Nouns/NewCustomTag</li>
<li>somethingelse->Words/Nouns/Something</li>
</CraftedArt.rulesFiles>


Though I'm still wondering if the existing key/paths should be translatable, or should they always remain the same. Replacing them with something completely different is a bit hacky and won't be needed after we add rulesFiles, so I'm not sure. On the other hand some languages may not be able to use them at all due to language grammar so they could as well be replaced with something useful. Furthermore if some translations already do this then they would still work.

Elevator

QuoteI think the best solution here would be to add a new field to rule packs: List<string> rulesFiles; and allow full-list translation.
The mechanism you've just described might be exactly the thing that will cover all the issues connected to file rules.
Let me check if I understand you correctly. The "Rule_File" rules will be moved to the specific section.

For example, file RulePacks_Global.xml like this:
<Defs>
  <RulePackDef>
    <defName>GlobalUtility</defName>
    <rulePack>
      <rulesStrings>
        <!-- ... -->
      </rulesStrings>
      <rulesRaw>
        <!-- ... -->
       
        <li Class="Rule_File" ListIndex="24">
          <keyword>mechanoid</keyword>
          <path>Words/Nouns/Mechanoid</path>
        </li>
        <li Class="Rule_File" ListIndex="25">
          <keyword>concept</keyword>
          <pathList>
            <li ListIndex="0">Words/Nouns/Concepts_Badass</li>
            <li ListIndex="1">Words/Nouns/Concepts_Positive</li>
            <li ListIndex="2">Words/Nouns/Concepts_Angsty</li>
          </pathList>
        </li>

        <!-- ... -->
      </rulesRaw>
    </rulePack>
  </RulePackDef>
</Defs>

will turn into something like this:
<Defs>
  <RulePackDef>
    <defName>GlobalUtility</defName>
    <rulePack>
      <rulesStrings>
        <!-- ... -->
      </rulesStrings>
      <rulesFiles>
        <!-- ... -->
       
        <!-- Single-file rule -->
        <li>mechanoid->Words/Nouns/Mechanoid</li>

        <!-- Multiple-file rule -->
        <li>concept->Words/Nouns/Concepts_Badass</li>
        <li>concept->Words/Nouns/Concepts_Positive</li>
        <li>concept->Words/Nouns/Concepts_Angsty</li>
       
        <!-- ... -->
      </rulesFiles>
      <rulesRaw>
        <!-- ... -->
      </rulesRaw>
    </rulePack>
  </RulePackDef>
</Defs>


If I get it right, that looks really elegant.
In some localizations projects (English, German) translators will not have to override <rulesFiles> section in DefInjections. In this case the original file paths will be used.
But for some other languages (like Russian, Ukranian and Polish) the translator may decide to override this section and achieve the great flexibility.
The hacky solution that I described in the OP (it is currently implemented in Russian translation) will no longer be needed, and can be safely processed by a script to fit new requrements.
So, the solution you proposed sounds very reasonable.

ison

I think that the Rule_Files will stay the same in defs. We'll just add a new field which adds extra file rules to the already existing ones. So all rules will be: rulesRaw + rulesStrings (instantiated as Rule_String) + rulesFiles (instantiated as Rule_File).

We could move all Rule_Strings to the new rulesFiles variable but then they will all get replaced if you do a full-list translation on them. So it won't be possible to e.g. add just 1 more file instead of replacing all of them.

(It's the same as with rulesStrings - they could be defined in the rulesRaw section, they have their own variable for convenience while translating rules)

Elevator

#10
I understand your idea but there are some points that would be nice to take into consideration

For example there is a rule in Defs/RulePackDefs/RulePacks_Global.xml:
<li Class="Rule_File">
  <keyword>AdjectiveAngsty</keyword>
  <path>Words/Adjectives/Angsty</path>
</li>

Some translator added custom file that contains the same adjectives, but for plural form:
<li>AdjectiveAngsty_plural->Words/Adjectives/Angsty_plural</li>
When this (or another) translator works on some rulepack, they have to seek for global keywords (AdjectiveAngsty or AdjectiveAngsty_plural) in two places (Defs/RulePackDefs/RulePacks_Global.xml and DefInjections/RulePackDefs/RulePacks_Global.xml) instead of one. Moreover, RulePacks_Global.xml is not the only context where file rules exist. A lot of things should be kept just in mind. This is a bit confusing and not very convenient.

If you choose to move all file rules from rulesRaw to rulesFiles this issue can is easily avoided, because all important things are in a single place (in Defs or in DefInjected):
<li>AdjectiveAngsty->Words/Adjectives/Angsty</li>
<li>AdjectiveAngsty_plural->Words/Adjectives/Angsty_plural</li>


The other point is that if a translator decides to make a full-list replacement of rulesFiles in DefInjections, they must know what they do and won't make foolish mistakes (e.g. replacaing all "default" file rules by one single useless line). If the way how this mechanism works is properly documented (for instance, in xml-comments in Def files) the risk of such mistakes will be minimal.

Well, anyway, with such a powerful tool as full-list translation of file rules it will be possible to achieve much better translation results than now.

Tynan

Quote from: ison on June 19, 2018, 11:13:34 AM
I think that the Rule_Files will stay the same in defs. We'll just add a new field which adds extra file rules to the already existing ones. So all rules will be: rulesRaw + rulesStrings (instantiated as Rule_String) + rulesFiles (instantiated as Rule_File).

That sounds smart; if you do go ahead with it please convert the existing XML to the shorthand system as well.
Tynan Sylvester - @TynanSylvester - Tynan's Blog

ison

#12
Alright, implemented the following:

- RulePacks have a new field rulesFiles which can be translated
- All RulePackDef use this new field instead of specyfing Rule_Files manually in rulesRaw
- All rules keys and paths are now MayTranslate instead of NoTranslate so you can translate them if you need

Implemented after 1.0.1940

btw. there's another feature coming where it'll be possible to do this:
<Gunshot.comps.HediffComp_GetsPermanent.permanentLabel>translation</Gunshot.comps.HediffComp_GetsPermanent.permanentLabel>
instead of this:
<Gunshot.comps.2.permanentLabel>translation</Gunshot.comps.2.permanentLabel>

so you don't have to remember all the indexes and the translations won't break if we decide to add one more element to the list

Elevator

Thank you for the update, ison.
I'll check these features out as soon as possible ).