support ICU, Fluent and other i18n message syntaxes

[ivanhofer]

typesafe-i18n currently uses it's own syntax for writing translation messages. This has a few reasons:

• easy to parse for computers:
  No overly complex code is neededd. typesafe-i18n is lightweight. The parser + runtime are less tahn 1kb small. Alone the parser for the ICU format is about 7.8kb big (https://bundlephobia.com/package/@formatjs/[email protected])
• easy to parse for humans:
  I (personally) find the ICU message syntax not so easy to read. Especially for more complex translations
• no other syntax (as far as I know) supports type information. It probably can be added to the syntax. But this would require to write a new parser.

But not all people may like or due to business reasons can use the typesafe-i18n syntax. By providing the option to choose a format, typesafe-i18n could e accessible to more users.

How this could work:

• first of all: bundle optimization needs to be implemented
• when bundling the ICU message format can be transformed to same object structure typesafe-i18n uses, so no ICU parser is needed on runtime. The code that is running in the browser doesn't care i what format the translations are written because it always get's the same object-structure.
• also the generator converts the ICU format to the same object-structure that is needed to generate the type information
• calling a translation would be the same for all message syntaxes
• adding typesafe parameters to the translations could be tricky, but this could be provided with: #104

[jbergstroem]

For me, the main win here is to be able to plug a site into services like https://crowdin.com. So, if it is a two-way translation layer (no pun intended) I guess you get best of two worlds.

[ivanhofer]

Yes, this is also the intention to be able to use existing i18n services.

If someone purely relies on a translation service, the message format that is stored in the repo doesn't really matter.
Maybe as a first step a message-format-converter could be implemented that converts ICU to the typeasfe-i18n syntax.
When importing the translations, the converter then could be used to store the translations on disk.

[artola]

@ivanhofer I am trying to integrate this library with Lokalise that might be the replacement for an old RMS that we have.

I did an export, and upload the file to Lokalise in 1.6 seconds 🥇

Now looking at the result (still not sure how to tweak one and the other) I find a "problem" with the select for example for gender.

Lokalise supports ICU messages: https://unicode-org.github.io/icu/userguide/format_parse/messages/#messageformat

For example like this: https://docs.lokalise.com/en/articles/5042246-application-resource-bundle-arb

What is your opinion here? Are you thinking in some kind of configuration to select the used format or an adapter pattern to apply conversions at import/export?

It might be better (if configured) to use ICU format, then a dev will write code in such way and if read the translations in the translations' provider also match. But probably you foresee some other edge cases.

[ivanhofer]

Hi @artola, typesafe-i18n uses it's own syntax. The reason behind this is:

1. no other format has the concept of types for parameters
2. the syntax is relatively easy to parse, hence the code stays small and performant

Like all decisions, there are always tradeoffs. In this case it's beeing incompatible with other formats. I haven't spent time yet on writing a converter for different formats. But my solution would be to have a process like this:

1. load translations from Lokalise
2. convert ICU to an object representation
   e.g. 'Hello {0}!' would become

   [
      { type: 'string', value: 'Hello ' },
      { type: 'variable', value: '0', type: 'unknown' },
      { type: 'string', value: '!' }
   ]

3. convert object representation to typesafe-i18n syntax
4. use the importer to store the translation on disk

And of course also the other way around to sync translations from code to Localise.

Each message syntax would require to have a "convert SYNTAX to object representation" and "convert object representation to SYNTAX". With this approach a lot of different syntaxes could be supported.

An open question remains what to do with concepts that are not supported in both syntaxes. But I do not expect this to be a major problem.

The solution above would be the easiest to implement, because the actual runtime could stay the same.
If typesafe-i18n would support reading ICU and other formats directly from disk, then It would also require to have a runtime that can parse that syntax, which would increase bundle size by a big amount. Or during build-time ICU could be converted to the typesafe-i18n syntax. You would probably also loose the good TypeScript support for your translations.

I am aware that the used syntax of typesafe-i18n could be a major blocker for adapting the library. I'm currently working on a bigger concept, I hopefully can share in a few weeks.

[ivanhofer]

The codebase already gets uglified at the end. But object properties can't really be transformed. If they were minified, someone consuming the output of that code would not able do do anything with it. If something changes anywhere in your code size, the object would probably get different keys each time you build the code.

It's not just bundle size, also memory usage improves slightly when having a smaller object. Something that does not matter if you just have a few objects, but if you load hundreds or thousands of translations, having an optimized code will be more important.

Therefore I have chosen to do it like this.

A new parser is needed that transforms the strings to a generic well structured object-structure.

The solution you describe in the last paragraph is part of the bigger concept I'm planning. As you already guessed right, by having a well defined structure, anyone would be able to replace certain parts of the i18n proccess if they don't fit their project's needs.

[osdiab]

Oh are you saying that because the code is generated, objects with this particular data structure are also written as-is to disk and therefore you end up with longer strings in the generated code that the user would call? Makes sense - though tbh my own build pipeline and anyone who’s using a modern framework would uglify that as well too anyway so I’d guess it still probably wouldn’t really matter :)

Anyway it’s irrelevant, still can make that kind of serializer/parser nonetheless. Looking forward to the reveal of your idea, sounds like it’s still time to wait on that object format you’re designing before the ICU parser stuff would be relevant to build. Perhaps for now I’ll research if there’s an ICU serializer somewhere/if I can’t find one then write one using the ICU parser’s types as a base in the meantime.

[ivanhofer]

though tbh my own build pipeline and anyone who’s using a modern framework would uglify that as well too

https://skalman.github.io/UglifyJS-online/

Object properties can't be uglified. You need to do it manually and that's what I'm doing by using single letter property names.

---

sounds like it’s still time to wait on that object format you’re designing before the ICU parser stuff would be relevant to build

Yes, we would need to define the object structure first. Then things could be parallelized.

---

Perhaps for now I’ll research if there’s an ICU serializer somewhere/if I can’t find one then write one using the ICU parser’s types as a base in the meantime.

Sounds great!

[ivanhofer]

@osdiab here it is: https://github.com/pipeli18ne/RFC
Please leave your feedback and let me know what you think about this concept.

[ivanhofer]

@osdiab I just have released version 5.20.0 which exposes two functions to convert a string from typesafe-i18n-syntax to an object-structure and the other way around. This should help you to make ICU compatible with typesafe-i18n.

import { experimentalParseMessage, experimentalSerializeMessage } from 'typesafe-i18n/parser'

Feedback welcome!