[Help wanted] Convert MMS models to sherpa-onnx

[csukuangfj]

Please see #890

@yaambe and @thewh1teagle have managed to convert MMS models to sherpa-onnx

There are plenty of MMS tts models from https://huggingface.co/facebook/

We need to

• Provide scripts to convert MMS models to sherpa-onnx
• Add them to our Android APKs

[thewh1teagle]

Add them to our Android APKs

Why are there so many APKs for each model? Wouldn't it be better to use a single APK per platform and provide links with Deep Links to models? A click on them would install the model in the app within a few seconds.

[csukuangfj]

A click on them would install the model in the app within a few seconds.

Would you mind contributing the code to implement such a click?

[thewh1teagle]

Would you mind contributing the code to implement such a click?

I can try
Does the library for Kotlin can enable all capabilities of sherpa-onnx so that single APK will support any model?
I'm trying to understand what will be the flow except for downloading the model.

Imagine a deep link such as ?action=download&model=url1&model=url2&model=url3 for multiple model eg. vad + speaker embedding.
After it downloaded, does the current app code should be able to use it easily?
In addition some models downloaded compress with tar or zip so there's need to use compression library.
Overall I think that in the long term it will simplify the repo of sherpa-onnx, and will save many build time.

[csukuangfj]

Does the library for Kotlin can enable all capabilities of sherpa-onnx so that single APK will support any model?

Yes. Kotlin APIs support all functions of sherpa-onnx.

---

By the way, our main goal is to provide easy to use APIs and the APKs we provide are to demonstrate how to use our provided APIs.

We try to make the code of the APK as easy to understand as possible.

Users can develop their own APP based on our example APKs.

[willwade]

Converted all to onnx - https://huggingface.co/willwade/mms-tts-multilingual-models-onnx/tree/main

[csukuangfj]

Amazing! There are 919 models!

I will try to add it to sherpa-onnx.

[willwade]

NB: Some I was seeing an error with the vits-mms script as it doesnt support uroman - thats the reason for less than the official MMS repo

[willwade]

NB: ive added this which could be useful: https://huggingface.co/willwade/mms-tts-multilingual-models-onnx/blob/main/languages-supported.json - (its really annoying the steps to get your language of choice if you dont know the three letter iso code. This file shows which ones we dont have onnx models for and which ones we have a sample.wav in the onnx directory for (its around 145 from memory)). It also has links to the hugging face directories for full model or onnx model.

[willwade]

Add them to our Android APKs

Why are there so many APKs for each model? Wouldn't it be better to use a single APK per platform and provide links with Deep Links to models? A click on them would install the model in the app within a few seconds.

I'd personally love this (I work for a charity by the way supporting people who cant speak - its a real need to have these languages available at a system wide level). The problem I have is kotlin/android is way out of my wheelhouse. We do need to do this for iOS and I'm sure I've seen someone do this for iOS too..

Having a system wide TTS system solves a ton of problems because a lot of apps dont have to embed or use any additional API. It just works to then use the system sdk for voice.

[studionexus-lk]

NB: ive added this which could be useful: https://huggingface.co/willwade/mms-tts-multilingual-models-onnx/blob/main/languages-supported.json - (its really annoying the steps to get your language of choice if you dont know the three letter iso code. This file shows which ones we dont have onnx models for and which ones we have a sample.wav in the onnx directory for (its around 145 from memory)). It also has links to the hugging face directories for full model or onnx model.

https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html

[willwade]

NB: ive added this which could be useful: https://huggingface.co/willwade/mms-tts-multilingual-models-onnx/blob/main/languages-supported.json - (its really annoying the steps to get your language of choice if you dont know the three letter iso code. This file shows which ones we dont have onnx models for and which ones we have a sample.wav in the onnx directory for (its around 145 from memory)). It also has links to the hugging face directories for full model or onnx model.

https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html

Nice! I might merge the data of that so its in a json format. I'll focus on the TTS column for this

[paolo-caroni]

By the way, our main goal is to provide easy to use APIs and the APKs we provide are to demonstrate how to use our provided APIs.

We try to make the code of the APK as easy to understand as possible.

Users can develop their own APP based on our example APKs.

We have discussed that on much issues, Android privide both TTS and STT API, a standard API to use with any app. I hope that some day sherpa-onnx will provide an engine that support completely such API. I doubt that someone would make an app that use specifically sherpa-onnx, instead of use a generic engine (that is not included on the app itself).

@thewh1teagle on #569 you can find a link to an experimental app that download TTS models, without the nonsense of have different apk package with same engine.

[studionexus-lk]

is this onnx models converted to sherpaonnx or we have to convert them?

[willwade]

is this onnx models converted to sherpaonnx or we have to convert them?

All been converted already https://huggingface.co/willwade/mms-tts-multilingual-models-onnx/

[willwade]

Re: APKs. I’m getting a lot of requests to build apks for tts for each of the 1300 mms models. Can someone point me in the direction of the build script for the apks ?

[csukuangfj]

Re: APKs. I’m getting a lot of requests to build apks for tts for each of the 1300 mms models. Can someone point me in the direction of the build script for the apks ?

Please see
https://github.com/k2-fsa/sherpa-onnx/blob/master/.github%2Fworkflows%2Fapk-tts-engine.yaml