|
--- |
|
title: VALL-E |
|
emoji: 🐻❄️ |
|
colorFrom: green |
|
colorTo: purple |
|
sdk: gradio |
|
sdk_version: 3.41.2 |
|
app_file: app.py |
|
pinned: false |
|
license: agpl-3.0 |
|
--- |
|
|
|
Recommended settings for stable speech: |
|
* `NAR levels`: 7; less NAR levels reduces the quality of the final waveform (this may also be strictly because, when EnCodec is fed a sequence with less RVQ bin levels than what it was initialized with, it'll sound worse). |
|
* `Temperature (AR)`: [0.85, 1.1]; It's ***really*** tough to find a one-size-fits-all value. |
|
* `Temperature (NAR)`: [0.15, 0.85]; This is even harder to nail decent values. Too high and you'll hear artifacts from the NAR, too low and you might not have acoustic detail recreated. |
|
* `Dynamic Temperature`: checked; Dynamic temperature seems to definitely help resolve issues with a model that is not strongly trained. Pairable with every other sampling technique. |
|
* `Top P`: [0.85, 0.95] || 1; I feel this is cope. |
|
* `Top K`: [768, 1024] || 0; I also feel this is cope. |
|
* `Beam Width`: 0 || 16; beam searching helps find potential best candidates, but I'm not sure how well it helps in the realm of audio. Incompatible with mirostat. |
|
* `Repetition Penalty`: 1.35; this and the length decay miraculously are what helps stabilize output; I have my theories. |
|
* `Repetition Penalty Length Decay`: 0.2; this helps not severly dampen the model's output when applying rep. pen. |
|
* `Length Penalty`: 0; this only be messed with if you're consistently having either too short output, or too long output. The AR is trained decently enough to know when to emit a STOP token. |
|
* `Mirostat (Tau)`: [2.0, 8.0]; The "surprise value" when performing mirostat sampling, which seems to be much more favorable in comparison to typical top-k/top-p or beam search sampling. The "best" values are still unknown. |
|
* `Mirostat (Eta)`: [0.05, 0.3]; The "learning rate" (decay value?) applied each step for mirostat sampling. |
|
|
|
This Space: |
|
* houses experimental models and the necessary inferencing code for my [VALL-E](https://git.ecker.tech/mrq/vall-e) implementation. I hope to gain some critical feedback with the outputs. |
|
* utilizes a T4 with a narcoleptic 5-minute sleep timer, as I do not have another system to (easily) host this myself with a 6800XT (or two) while I'm training off my 4070Ti and 7900XTX. |
|
|
|
The model is: |
|
* utilizing an RetNet for faster training/inferencing with conforming dimensionality (1024 dim, 4096 ffn dim, 16 heads, 12 layers) targetting the full eight RVQ-bins (albeit the model was originally trained at two then four). |
|
* trained on ~12.5K hour dataset composed of LibriTTS-R, LibriLight (`small`+`medium`+`duplicated`), generously donated audiobooks, and vidya voice clip rips (including some Japanese kusoge gacha clips). |
|
* a "monolothic" approach to sharing the retention-based transformer weights between AR and NAR tasks for no immediately discernable penalties (besides retraining). |
|
* utilizing DeepSpeed to inference using its int8 quantized inferencing (allegedly), and Vocos for better output quality. |
|
- I do need to add a toggle between different dtypes to gauge any perceptable quality/throughput gains/losses. |
|
* currently still being trained, and any updates to it will be pushed back to this repo. |
|
|
|
I am also currently training an experimental model with double the layers (24 layers instead) to gauge its performance. Depending on how well it performs, I may pivot to that too, but for now, I'm starting to doubt the time investment in training it. |
|
|
