🥑 alVoCat
🥑 alVoCat is a vocoder for Catalan TTS, based on Vocos architecture. It is highly performant and high quality, works together with 🍵 Matxa and you can find our fork here and a demo here.
Model Details
Model Description
Vocos is a fast neural vocoder designed to synthesize audio waveforms from acoustic features. Unlike other typical GAN-based vocoders, Vocos does not model audio samples in the time domain. Instead, it generates spectral coefficients, facilitating rapid audio reconstruction through inverse Fourier transform.
This version of Vocos uses 80-bin mel spectrograms as acoustic features which are widespread in the TTS domain since the introduction of hifi-gan The goal of this model is to provide an alternative to hifi-gan that is faster and compatible with the acoustic output of several TTS models. This version is tailored for the Catalan language, as it was trained only on Catalan speech datasets.
We are grateful with the authors for open sourcing the code allowing us to modify and train this version.
Intended Uses and limitations
The model is aimed to serve as a vocoder to synthesize audio waveforms from mel spectrograms. Is trained to generate speech and if is used in other audio domain is possible that the model won't produce high quality samples.
How to Get Started with the Model
Use the code below to get started with the model.
Installation
To use Vocos only in inference mode, install it using:
pip install git+https://github.com/langtech-bsc/vocos.git@matcha
Reconstruct audio from mel-spectrogram
import torch
from vocos import Vocos
vocos = Vocos.from_pretrained("projecte-aina/alvocat-vocos-22khz")
mel = torch.randn(1, 80, 256) # B, C, T
audio = vocos.decode(mel)
Copy-synthesis from a file:
import torchaudio
y, sr = torchaudio.load(YOUR_AUDIO_FILE)
if y.size(0) > 1: # mix to mono
y = y.mean(dim=0, keepdim=True)
y = torchaudio.functional.resample(y, orig_freq=sr, new_freq=22050)
y_hat = vocos(y)
Onnx
We also release an onnx version of the model, you can check in colab:
Training Details
Training Data
The model was trained on 3 Catalan speech datasets
Dataset | Language | Hours |
---|---|---|
Festcat | ca | 22 |
OpenSLR69 | ca | 5 |
LaFrescat | ca | 3.5 |
Training Procedure
The model was trained for 1.5M steps and 1.3k epochs with a batch size of 16 for stability. We used a Cosine scheduler with an initial learning rate of 5e-4. We also modified the mel spectrogram loss to use 128 bins and fmax of 11025 instead of the same input mel spectrogram.
Training Hyperparameters
- initial_learning_rate: 5e-4
- scheduler: cosine without warmup or restarts
- mel_loss_coeff: 45
- mrd_loss_coeff: 0.1
- batch_size: 16
- num_samples: 16384
Evaluation
Evaluation was done using the metrics on the original repo, after ~ 1000 epochs we achieve:
- val_loss: 3.57
- f1_score: 0.95
- mel_loss: 0.22
- periodicity_loss: 0.113
- pesq_score: 3.31
- pitch_loss: 31.61
- utmos_score: 3.33
Citation
If this code contributes to your research, please cite the work:
@article{siuzdak2023vocos,
title={Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis},
author={Siuzdak, Hubert},
journal={arXiv preprint arXiv:2306.00814},
year={2023}
}
Additional information
Author
The Language Technologies Unit from Barcelona Supercomputing Center.
Contact
For further information, please send an email to langtech@bsc.es.
Copyright
Copyright(c) 2024 by Language Technologies Unit, Barcelona Supercomputing Center.
License
Creative Commons Attribution Non-commercial 4.0
These models are free to use for non-commercial and research purposes. Commercial use is only possible through licensing by the voice artists. For further information, contact langtech@bsc.es and lafrescaproduccions@gmail.com.
Funding
This work has been promoted and financed by the Generalitat de Catalunya through the Aina project.
Part of the training of the model was possible thanks to the compute time given by Galician Supercomputing Center CESGA (Centro de Supercomputación de Galicia)
- Downloads last month
- 36