Published AI Model for ChatTTS Speech Synthesis

Project ChatTTS published model and related machine learning tools for emotional speech synthesis. The ChatTTS project is specifically optimized for use in conversational systems, such as interactive assistants, and aims to reproduce the properties of natural emotional communication. Interaction with multiple speakers and building an interactive dialogue are supported. Correctly tracked and reproduced during synthesis prosodic elementssuch as laughter, pauses and interjections.

When training the model, about 40 thousand hours of speech recordings were used (in the non-public version of the model – 100 thousand hours). According to the developers, the model’s intonation generation capabilities are superior to all previously available open speech synthesis models. To control emotions during synthesis, only token substitution is currently supported, for example, “(laugh)” for laughter. To generate a 30-second recording, a GPU with 4 GB of memory is required. On the NVIDIA GeForce RTX 4090D GPU, the generation rate is approximately 7 semantic tokens per second. Synthesis by female and male voices in English and Chinese is supported (for the Russian language we can recommend the framework TTS and model XTTS-v2which in addition to synthesis support voice cloning from a short speech recording, including for synthesis in another language).

The ChatTTS model is published under a CC BY-NC-ND 4.0 (Creative Commons Attribution-NonCommercial-NoDerivatives 4.0) license, which allows free redistribution with attribution, but prohibits the creation of derivative works and use in commercial projects. In addition, to protect against the use of the model to commit fraudulent and criminal acts, when training the model, substitution of high-frequency noise was used and the maximum level of audio compression was used using the MP3 format.

Thanks for reading: