stuff

README.md

@@ -10,7 +10,7 @@ To reiterate, this is ***by no means*** complete. I am not passing this off as c
 
 ## Models
 
-This repo contains the following configurations:
+This repo contains the following configurations under `./models/`:
 
 * `config.retnet.yaml` / `ar+nar-retnet-8`: The previously released weights.
 	+ This configuration utilizes a RetNet (retention based "transformer") as the underlying architecture due to a number of misleading interpretations with comparisons, for better or for worse.
@@ -34,20 +34,26 @@ This repo contains the following configurations:
 	+ ~~However, it seems to *only* do well with long utterances. Short utterances fumble. I believe further training with a variety of durations should allow the AR to handle a variety of durations.~~
 		- ~~I believe the "slowly stepping up the context length" only works for text, and not audio.~~
         - Addendum: Additional brief training for a variety of duration lengths seemed to have mostly fixed this issue.
+        - Addendum addendum: Properly creating the position IDs per-segment rather than the whole sequence, also helps a lot.
 	+ Zero-shot performance leaves a bit to be desired, as it did not receive the special training prioritizing shuffling between speakers rather than the global pool of utterances.
         - Addendum: Additional brief training for sampling based on speaker per "epoch" (per dataloader, not dataset) seemed to slightly improve it.
 	+ Testing showed that, despite also stepping up the prompt duration, it *really* likes three second prompts.
 	+ Definitely needs additional training, but the next way to go is unknown.
         + Naturally, training it on a "next RVQ level is half as likely" distribution introduces some crust as the later RVQ levels are less accurate, introducing noise and artifacts.
-        + Naively training it on equally distributed RVQ levels *does* lobotomize the AR.
+        + As a fix for the above, naively training it on equally distributed RVQ levels *does* lobotomize the AR.
         + Additional training on the AR will see huge diminishing returns, so I don't know if it's worth doing so.
     + Seems to be a decent foundation for "distillation", at the very least for LoRA training.
+    	- Addendum: it seems to serve fine for patch-training a few extra tweaks, to non-unified position IDs, split classifier heads, and para-parallel decoding for the AR.
+
+## Experiments
+
+Under `./models/experiments/` are some failed models, but are included to serve as references for my errors. Do ***not*** use them unless you're curious, or know what you're doing.
 
 * `config.llama.split.yaml` / `ar-llama-1` + `nar-llama-8`: The above model, but split and trained a little bit more.
 	+ This experiment is to see whether the AR and NAR benefitted from being split up after enough pretraining, to un-"lobotomize" any penalties from attending to two different tasks (as the AR predicts the next token, and the NAR predicts the same token but a different level).
 	+ I believe I trained each separate model an additional extra day for another additional audio-duration window for similar training lengths.
 	+ ~~I don't think audio quality differs a non-trivial amount to warrant splitting the model.~~
-        - From recent experiments, it does seem a NAR-only model is beneficial.
+        - Addendum: From recent experiments, it does seem a NAR-only model is beneficial; I will need to explore this in the future.
 
 * `config.dac.yaml` / `ar+nar-dac-llama-9`: Utilizes [Descript-Audio-Codec](https://github.com/descriptinc/descript-audio-codec/) instead as the audio backend.
     + This utilizies the 44KHz (erroneously at 44,000 Hz instead of 44,100 Hz) model at 9 RVQ levels (majorly trained at 8, then the 9th was included).
@@ -55,20 +61,29 @@ This repo contains the following configurations:
         + Later experimented with the 24Khz model, but training would *always* diverge.
     + *Heavily* benefits from inferencing only the first four RVQ levels; levels afterwards includes far too much noise in the final output.
         + I imagine the nature of DAC itself amplifies errors in the remaining RVQ levels (either due to less resilliency to errors in the codes, or each RVQ level affecting hte final waveform more).
+        + Addendum: restricting to the first four RVQ levels seems to help remove noisy artifacts, but quality is hindered as there's still less RVQ levels to rely on.
     + Has not received as much training as the EnCodec-based models.
         + Because of this, performance leaves more to be desired.
     + Further experimentation is needed, but the next approach is unknown.
         + Train a NAR only model to help bolster the remaining RVQ levels (outputted utterances seem a bit sluggish).
         + Continue training the AR+NAR to try and bolster the AR tasks (as it's quite lacking at the moment).
         + Delve into other, exotic features, such as utilizing DAC's decoding embeddings (which might not be necessary at all since it seems *fine* at the moment).
+        	+ Addendum: This seems unneccessary, as freezing to these embeddings is harmful, and not freezing them will just inevitably cause them to shift elsewhere.
 
-Some additional configurations have been explored with, but experiments have not been fruitful:
+* `config.dac-nar-len.yaml` / `nar-len-llama-9`: A DAC-based model, but is a pure NAR model (+ autoregressive length task) .
+	+ Originally thought to be bunk from inferencing tests having audio drastically drop off into silence, but I suppose it was just some issue that eventually resolved itself.
+	+ Suffers from the same problems the above model suffers from (terrible quality).
+	+ *Huge* performance gains, but may definitely suffer from some specific qualities in the outputs, if it does get trained right.
 
-* Exotic wrappers like `BitNet` seemed to yield little gains in inferencing, somehow.
+* `config.llama-x4.yaml` / `ar+nar-llama-8`: The above `ar+nar-llama-8` model, but with para-parallel decoding for the AR in-post.
+	+ This mostly serves as a proof-of-concept for speeding up inferencing by reducing the number of steps required, by decoding multiple tokens in parallel with a similar approach to how the NAR decodes in parallel.
+		+ Trained with the trainer's batch-by-durations sampler for a maximum duration batch size of 100 seconds (750 resp tokens), with ProdigyOpt at bfloat16 (no AMP) on my 4070Ti (because I can't be assed to fire up my 4xV100 machine again for a simple test).
+	+ The model definitely needs to be retrained as there's some errors for the additional tokens.
+		+ If these cannot be nailed out with more training, then I imagine a similar approach to speculative decoding where the nth tokens are discarded if the confidence is low.
+		+ Greedy sampling might be beneficial instead for this, as the NAR does benefit greatly from low temperatures / greedy sampling.
+
+Some additional configurations have been explored with, but experiments have not been fruitful:
 
-* Mamba / Mamba2-based models have shown that it's ***really*** hard to have an AR+NAR model.
+* Exotic wrappers like `BitNet` seemed to yield little gains in inferencing, somehow. The memory savings is pretty much unneccessary as the models are already manageable at ~200M parameters.
 
-* A NAR only model has been experimented with, but seemed utterly useless in practice.
-    + The underlying architecture will query the model for the duration, and then inference *all* RVQ levels in parallel (one level at a time).
-    + Despite working in the overfitting test trainer and decent training metrics, inferencing will have the model fall completely flat.
-    + I have zero ideas for which path to go with for further experimentation.
+* Mamba / Mamba2-based models have shown that it's ***really*** hard to have an AR+NAR model. I really do not want to bother throwing the compute at another ~~meme~~ arch I can't easily make use of all the other tech to throw at.

models/experiments/config.dac-nar-len.yaml

@@ -0,0 +1,133 @@
+sample_rate: 44_000
+audio_backend: "dac"
+
+models:
+- name: "nar-len"
+  size:
+    audio_tokens: 1024
+    text_tokens: 256
+    dim: 1024
+    heads: 16
+    layers: 16
+  resp_levels: 9
+  prom_levels: 9
+  tasks: 8
+  langs: 2
+  tones: 1
+  arch_type: llama
+  training: True
+  version: 5
+  attention: flash_attention_2
+  dropout: 0.1
+  #loss_factors:
+  #  text: 0.01
+  #  prom: 0.5
+  #  resp: 1.0
+  #  len: 1.0
+  capabilities: ["nar", "len"]
+  experimental:
+    audio_embedding_sums: False
+    interleave: False
+    unified_position_ids: True
+    rvq_level_range: []
+    split_classifiers: True
+    tie_classifier_to_embedding: False
+
+#loras:
+#- name : "lora-test"
+#  rank: 128
+#  alpha: 128
+#  training: True
+#  rvq_levels: []
+
+hyperparameters:
+  batch_size: 16
+  gradient_accumulation_steps: 4
+  gradient_clipping: 1.0
+  warmup_steps: 10
+
+  optimizer: Prodigy
+  learning_rate: 1.0
+  torch_optimizer: True
+  
+  scheduler: "" # ScheduleFree
+  torch_scheduler: True
+
+evaluation:
+  batch_size: 4
+  frequency: 250
+  size: 4
+  
+  steps: 500
+  ar_temperature: 1.0
+  nar_temperature: 0.0
+
+trainer:
+  iterations: 1_000_000  
+  save_frequency: 250
+  keep_last_checkpoints: 4
+  
+  check_for_oom: False
+  gradient_checkpointing: False
+
+  weight_dtype: bfloat16
+  amp: False
+
+  backend: deepspeed
+  deepspeed:
+    inferencing: False
+    amp: False
+
+  load_webui: False
+
+inference:
+  backend: local
+  normalize: False
+
+  weight_dtype: bfloat16
+  amp: False
+
+optimizations:
+  injects: False
+  replace: True
+
+  linear: False
+  embedding: False
+  optimizers: True
+
+  bitsandbytes: False
+  dadaptation: False
+  bitnet: False
+  fp8: False
+
+dataset:
+  speaker_name_getter: "lambda p: f'{p.parts[-3]}_{p.parts[-2]}'"
+  speaker_group_getter: "lambda p: f'{p.parts[-3]}'"
+
+  use_hdf5: True
+  hdf5_flag: r
+  
+  use_metadata: True
+  validate: True
+
+  workers: 1
+  cache: False
+
+  duration_range: [3.0, 24.0]
+
+  random_utterance: 1.0
+  max_prompts: 1
+  prompt_duration_range: [3.0, 3.0]
+  
+  max_resps: 1
+  p_resp_append: 0.25
+
+  sample_type: path # path # speaker
+  sample_order: duration
+  sample_max_duration_batch: 100
+
+  tasks_list: [ "tts" ] #, "tts-c", "ns", "sr" ]
+
+  training: []
+  validation: []
+  noise: []

old/.cache.tar.gz

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:993b57269f1d012c9f5b46372a1f2320e75872da177f7b2071a38578a2ca166b
-size 16020894

old/.gitattributes

@@ -1,35 +0,0 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

old/data.tar.gz

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6b22eed2c03d206a951f5d20db8c0772eb3d49ee2c7b6483bb035bcd240d1971
-size 42633974681