Init

updated README.md

README.md

@@ -1,12 +1,39 @@
 ---
-title: Vall E
-emoji: 🔥
-colorFrom: yellow
+title: VALL-E
+emoji: 🐻‍❄️
+colorFrom: green
 colorTo: purple
 sdk: gradio
 sdk_version: 3.41.2
 app_file: app.py
 pinned: false
+license: agpl-3.0
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+Recommended settings for stable speech:
+* `NAR levels`: 7; less NAR levels reduces the quality of the final waveform (this may also be strictly because, when EnCodec is fed a sequence with less RVQ bin levels than what it was initialized with, it'll sound worse).
+* `Temperature (AR)`: [0.85, 1.1]; It's ***really*** tough to find a one-size-fits-all value.
+* `Temperature (NAR)`: [0.15, 0.85]; This is even harder to nail decent values. Too high and you'll hear artifacts from the NAR, too low and you might not have acoustic detail recreated.
+* `Dynamic Temperature`: checked; Dynamic temperature seems to definitely help resolve issues with a model that is not strongly trained. Pairable with every other sampling technique.
+* `Top P`: [0.85, 0.95] || 1; I feel this is cope.
+* `Top K`: [768, 1024] || 0; I also feel this is cope.
+* `Beam Width`: 0 || 16; beam searching helps find potential best candidates, but I'm not sure how well it helps in the realm of audio. Incompatible with mirostat.
+* `Repetition Penalty`: 1.35; this and the length decay miraculously are what helps stabilize output; I have my theories.
+* `Repetition Penalty Length Decay`: 0.2; this helps not severly dampen the model's output when applying rep. pen.
+* `Length Penalty`: 0; this only be messed with if you're consistently having either too short output, or too long output. The AR is trained decently enough to know when to emit a STOP token.
+* `Mirostat (Tau)`: [2.0, 8.0]; The "surprise value" when performing mirostat sampling, which seems to be much more favorable in comparison to typical top-k/top-p or beam search sampling. The "best" values are still unknown.
+* `Mirostat (Eta)`: [0.05, 0.3]; The "learning rate" (decay value?) applied each step for mirostat sampling.
+
+This Space:
+* houses experimental models and the necessary inferencing code for my [VALL-E](https://git.ecker.tech/mrq/vall-e) implementation. I hope to gain some critical feedback with the outputs.
+* utilizes a T4 with a narcoleptic 5-minute sleep timer, as I do not have another system to (easily) host this myself with a 6800XT (or two) while I'm training off my 4070Ti and 7900XTX.
+
+The model is:
+* utilizing an RetNet for faster training/inferencing with conforming dimensionality (1024 dim, 4096 ffn dim, 16 heads, 12 layers) targetting the full eight RVQ-bins (albeit the model was originally trained at two then four).
+* trained on ~12.5K hour dataset composed of LibriTTS-R, LibriLight (`small`+`medium`+`duplicated`), generously donated audiobooks, and vidya voice clip rips (including some Japanese kusoge gacha clips).
+* a "monolothic" approach to sharing the retention-based transformer weights between AR and NAR tasks for no immediately discernable penalties (besides retraining).
+* utilizing DeepSpeed to inference using its int8 quantized inferencing (allegedly), and Vocos for better output quality.
+    - I do need to add a toggle between different dtypes to gauge any perceptable quality/throughput gains/losses.
+* currently still being trained, and any updates to it will be pushed back to this repo.
+
+I am also currently training an experimental model with double the layers (24 layers instead) to gauge its performance. Depending on how well it performs, I may pivot to that too, but for now, I'm starting to doubt the time investment in training it.

app.py

@@ -0,0 +1,5 @@
+import sys
+
+sys.argv = sys.argv + "--dtype=int8 --device=cuda".split(" ")
+
+from vall_e.webui import ui

models/ckpt/ar+nar-retnet-4/fp32.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91781616d713a424cab977abb718888323d1a26461bef78c8065ac30d1258d2a
+size 424338659

models/ckpt/ar+nar-retnet-8/fp32.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82db6c749d682f84881f8ec8aa3b402b56ce63118867be46bf9dada34dc0ded5
+size 441076031

models/ckpt/ar-retnet-4/fp32.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e042d05f14f21a166cd5f5c16b9c9c4ac9ce18af2a4c285c7f0d3ef3ea6729bf
+size 418040575

models/ckpt/nar-retnet-4/fp32.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18027cafe3c077cb8786a5665f04f732f4e3fcacff17844182f9383a1dca640f
+size 422230719

models/config.ar_nar.yaml

@@ -0,0 +1,126 @@
+dataset:
+  training: []
+  validation: [  ]
+  noise: []
+  
+  speaker_name_getter: "lambda p: f'{p.parts[-3]}_{p.parts[-2]}'"
+  
+  use_hdf5: True
+  use_metadata: True
+  hdf5_flag: r
+  validate: True
+
+  workers: 2
+  cache: True
+
+  phones_range: [4, 256]
+  duration_range: [1.0, 16.0]
+
+  random_utterance: 1.0
+  max_prompts: 3
+  prompt_duration: 6.0
+
+  sample_type: speaker
+
+  tasks_list: [ "tts" ] # , [ "tts", "tts-c", "ns", "sr", "tse", "cse", "nse", "tts"]
+
+models:
+  _prom_levels: 8
+  _max_levels: 8
+
+  _models:
+  - name: "ar+nar"
+    size: "full"
+    resp_levels: 8
+    prom_levels: 8
+    tasks: 8
+    langs: 2
+    arch_type: "retnet"
+    training: True
+    version: 3
+
+hyperparameters:
+  batch_size: 8
+  gradient_accumulation_steps: 32
+  gradient_clipping: 100
+  
+  optimizer: AdamW # Prodigy
+  torch_optimizer: True
+  learning_rate: 1.0e-4
+  
+  scheduler_type: ""
+  #scheduler_type: OneCycle
+  #scheduler_params:
+  #  cycle_first_step_size: 10_000
+  #  cycle_first_stair_count: 10_000
+
+  #  cycle_second_step_size: 15_000
+  #  cycle_second_stair_count: 15_000
+
+  #  decay_step_size: 5_000
+
+  #  cycle_min_lr: 2.5e-4 # 1.0e-5
+  #  cycle_max_lr: 2.5e-4 # 1.0e-4
+  #  decay_lr_rate: 0.0
+
+  #  cycle_min_mom: 0.90
+  #  cycle_max_mom: 0.99
+  #  decay_mom_rate: 0.0
+
+evaluation:
+  batch_size: 16
+  frequency: 250
+  size: 16
+  
+  steps: 450
+  ar_temperature: 0.95
+  nar_temperature: 0.25
+  load_disabled_engines: True
+
+trainer:
+  iterations: 1_000_000
+  
+  save_tag: step
+  save_on_oom: True
+  save_on_quit: True
+  save_frequency: 100
+  export_on_save: True
+
+  keep_last_checkpoints: 4
+
+  aggressive_optimizations: False
+  load_disabled_engines: False
+
+  load_state_dict: True
+  #strict_loading: False
+  #load_tag: "9500"
+  #load_states: False
+  #restart_step_count: True
+  
+  gc_mode: None # "global_step"
+
+  weight_dtype: bfloat16
+  amp: False
+
+  backend: deepspeed
+  deepspeed:
+    inferencing: True
+    zero_optimization_level: 0
+    use_compression_training: False
+
+  activation_checkpointing: True
+
+inference:
+  backend: local
+  use_vocos: True
+  normalize: False
+
+  weight_dtype: bfloat16
+  amp: False
+
+bitsandbytes:
+  enabled: False
+  injects: True
+  linear: True
+  embedding: True
+  

models/config.yaml

@@ -0,0 +1,104 @@
+dataset:
+  training: [
+  ]
+  validation: [
+  ]
+  noise: [
+  ]
+  
+  speaker_name_getter: "lambda p: f'{p.parts[-3]}_{p.parts[-2]}'"
+  
+  use_hdf5: True
+  use_metadata: True
+  hdf5_flag: r
+  validate: True
+
+  workers: 4
+  cache: True
+
+  phones_range: [4, 256]
+  duration_range: [1.0, 16.0]
+
+  random_utterance: 1.0
+  max_prompts: 3
+  prompt_duration: 3.0
+
+  sample_type: speaker
+
+  tasks_list: ["tts"] # , "ns", "sr", "tse", "cse", "nse", "tts"]
+
+models:
+  _prom_levels: 4
+  _max_levels: 4
+
+  _models:
+  - name: "ar"
+    size: "full"
+    resp_levels: 1
+    prom_levels: 2
+    tasks: 8
+    arch_type: "retnet"
+    training: True
+  - name: "nar"
+    size: "full"
+    resp_levels: 3
+    prom_levels: 4
+    tasks: 8
+    arch_type: "retnet"
+    training: True
+
+
+hyperparameters:
+  batch_size: 8
+  gradient_accumulation_steps: 1
+  gradient_clipping: 100
+  
+  optimizer: AdamW
+  learning_rate: 1.0e-5
+  
+  scheduler_type: ""
+
+evaluation:
+  batch_size: 16
+  frequency: 500
+  size: 16
+  
+  steps: 300
+  ar_temperature: 0.95
+  nar_temperature: 0.25
+  load_disabled_engines: True
+
+trainer:
+  iterations: 1_000_000
+  
+  save_tag: step
+  save_on_oom: True
+  save_on_quit: True
+  save_frequency: 100
+  export_on_save: True
+
+  keep_last_checkpoints: 4
+
+  load_state_dict: True
+
+  gc_mode: None # "global_step"
+
+  weight_dtype: bfloat16
+  amp: False
+
+  backend: deepspeed
+  deepspeed:
+    zero_optimization_level: 0
+    use_compression_training: True
+
+  activation_checkpointing: True
+
+inference:
+  use_vocos: True
+  normalize: False
+
+bitsandbytes:
+  enabled: False
+  injects: False
+  linear: False
+  embedding: False

packages.txt

@@ -0,0 +1 @@
+espeak-ng

requirements.txt

@@ -0,0 +1,8 @@
+--extra-index-url https://download.pytorch.org/whl/cu118
+torch
+torchaudio
+
+
+deepspeed==0.10.0
+vall_e @ git+https://git.ecker.tech/mrq/vall-e
+transformers