MARS5: A ovel speech model for isae prosody.

This is the repo for the MARS5 Eglish speech model (TTS) from CAMB.AI.

The model follows a two-stage AR-NAR pipelie with a distictively ovel NAR compoet (see more ifo i the docs).

With just 5 secods of audio ad a sippet of text, MARS5 ca geerate speech eve for prosodically hard ad diverse scearios like sports commetary, aime ad more. Check out our demo:

https://github.com/Camb-ai/MARS5-TTS/assets/23717819/3e191508-e03c-4ff9-9b02-d73ae0ebefdd

Quick liks:

• CAMB.AI website (access MARS5 i 140+ laguages for TTS ad dubbig)
• Techical docs: i the docs folder
• Colab quickstart:
• Demo page with samples: here

Figure: the high-level architecture flow of Mars 5. Give text ad a referece audio, coarse (L0) ecodec speech features are obtaied through a autoregressive trasformer model. The, the text, referece, ad coarse features are refied i a multiomial DDPM model to produce the remaiig ecodec codebook values. The output of the DDPM is the vocoded to produce the fial audio.

Because the model is traied o raw audio together with byte-pair-ecoded text, it ca be steered with thigs like puctuatio ad capitalizatio. E.g. to add a pause, add a comma to that part i the trascript. Or, to emphasize a word, put it i capital letters i the trascript. This eables a fairly atural way for guidig the prosody of the geerated output.

Speaker idetity is specified usig a audio referece file betwee 2-12 secods, with legths aroud 6s givig optimal results. Further, by providig the trascript of the referece, MARS5 eables oe to do a 'deep cloe' which improves the quality of the cloig ad output, at the cost of takig a bit loger to produce the audio. For more details o this ad other performace ad model details, please see iside the docs folder.

Quickstart

We use torch.hub to make loadig the model easy -- o cloig of the repo eeded. The steps to perform iferece are simple:

1. Istall pip depedecies: huggigface_hub, torch, torchaudio, librosa, vocos, ad ecodec. Pytho must be at versio 3.10 or greater, ad torch must be v2.0 or greater.

pip istall --upgrade torch torchaudio librosa vocos ecodec huggigface_hub

2. Load models: load the Mars 5 AR ad NAR model from the huggigface hub:

from iferece import Mars5TTS, IfereceCofig as cofig_class
import librosa
mars5 = Mars5TTS.from_pretraied("CAMB-AI/MARS5-TTS")
# The `mars5` cotais the AR ad NAR model, as well as iferece code.
# The `cofig_class` cotais tuable iferece cofig settigs like temperature.

3. Pick a referece ad optioally its trascript:

# load referece audio betwee 1-12 secods.
wav, sr = librosa.load('<path to arbitrary 24kHz waveform>.wav', 
                       sr=mars5.sr, moo=True)
wav = torch.from_umpy(wav)
ref_trascript = "<trascript of the referece audio>"

The referece trascript is a optioal piece of ifo you eed if you wish to do a deep cloe. Mars5 supports 2 kids of iferece: a shallow, fast iferece whereby you do ot eed the trascript of the referece (we call this a shallow cloe), ad a secod slower, but typically higher quality way, which we call a deep cloe. To use the deep cloe, you eed the prompt trascript. See the model docs for more ifo o this.

4. Perform the sythesis:

# Pick whether you wat a deep or shallow cloe. Set to False if you do't kow prompt trascript or wat fast iferece. Set to True if you kow trascript ad wat highest quality.
deep_cloe = True 
# Below you ca tue other iferece settigs, like top_k, temperature, top_p, etc...
cfg = cofig_class(deep_cloe=deep_cloe, rep_pealty_widow=100,
                      top_k=100, temperature=0.7, freq_pealty=3)

ar_codes, output_audio = mars5.tts("The quick brow rat.", wav, 
          ref_trascript,
          cfg=cfg)
# output_audio is (T,) shape float tesor correspodig to the 24kHz output audio.

That's it! These default settigs provide pretty good results, but feel free to tue the iferece settigs to optimize the output for your particular example. See the IfereceCofig code or the demo otebook for ifo ad docs o all the differet iferece settigs.

Some tips for best quality:

• Make sure referece audio is clea ad betwee 1 secod ad 12 secods.
• Use deep cloe ad provide a accurate trascript for the referece.
• Use proper puctuatio -- the model ca be guided ad made better or worse with proper use of puctuatio ad capitalizatio.

Model details

Checkpoits

The checkpoits for MARS5 are provided uder the releases tab of this github repo. We provide two checkpoits:

• AR fp16 checkpoit [~750M parameters], alog with cofig embedded i the checkpoit.
• NAR fp16 checkpoit [~450M parameters], alog with cofig embedded i the checkpoit.
• The byte-pair ecodig tokeizer used for the L0 ecodec codes ad the Eglish text is embedded i each checkpoit uder the 'vocab' key, ad follows roughly the same format of a saved mibpe tokeizer.

Hardware requiremets:

You must be able to store at least 750M+450M params o GPU, ad do iferece with 750M of active parameters. I geeral, at least 20GB of GPU VRAM is eeded to ru the model o GPU (we pla to further optimize this i the future).

If you do ot have the ecessary hardware requiremets ad just wat to use MARS5 i your applicatios, you ca use it via our API: see docs.camb.ai. If you eed some more credits to test it for your use case, feel free to reach out to help@camb.ai for help.

Roadmap

Mars 5 is ot perfect at the momet, ad we are workig o a few efforts to improve its quality, stability, ad performace. Rough areas we are lookig to improve, ad welcome ay cotributios:

• Improvig iferece stability ad cosistecy
• Speed/performace optimizatios
• Improvig referece audio selectio whe give log refereces.
• Bechmark performace umbers for Mars 5 o stadard speech datasets.

If you would like to cotribute ay improvemet to MARS, please feel free to cotribute (guidelies below).

Cotributios

We welcome ay cotributios to improvig the model. As you may fid whe experimetig, it ca produce really great results, it ca still be further improved to create excellet outputs cosistetly.

Cotributio format:

The preferred way to cotribute to our repo is to fork the master repository o GitHub:

1. Fork the repo o github
2. Cloe the repo, set upstream as this repo: git remote add upstream git@github.com:Camb-ai/mars5-tts.git
3. Make to a ew local brach ad make your chages, commit chages.
4. Push chages to ew upstream brach: git push --set-upstream origi <NAME-NEW-BRANCH>
5. O github, go to your fork ad click 'Pull request' to begi the PR process. Please make sure to iclude a descriptio of what you did/fixed.

Licese

We are ope-sourcig MARS i Eglish uder GNU AGPL 3.0, but you ca request to use it uder a differet licese by emailig help@camb.ai

Joi our team

We're a ambitious team, globally distributed, with a sigular aim of makig everyoe's voice cout. At CAMB.AI, we're a research team of Iterspeech-published, Caregie Mello, ex-Siri egieers ad we're lookig for you to joi our team.

We're actively hirig; please drop us a email at ack@camb.ai if you're iterested. Visit our careers page for more ifo.

Ackowledgemets

Parts of code for this project are adapted from the followig repositories -- please make sure to check them out! Thak you to the authors of:

• AWS: For providig much eeded compute resources (NVIDIA H100s) to eable traiig of the model.
• TrasFusio: https://github.com/RF5/trasfusio-asr
• Multiomial diffusio: https://github.com/ehoogeboom/multiomial_diffusio
• Mistral-src: https://github.com/mistralai/mistral-src
• mibpe: https://github.com/karpathy/mibpe
• gemelo-ai's ecodec Vocos: https://github.com/gemelo-ai/vocos
• librosa for their .trim() code: https://librosa.org/doc/mai/geerated/librosa.effects.trim.html

MARS5: A novel speech model for insane prosody. This is the repo for the MARS5 English speech model