Update README.md

README.md

@@ -72,6 +72,7 @@ Under `./models/experiments/` are some failed models, but are included to serve
 
 * `config.dac-nar-len.yaml` / `nar-len-llama-9`: A DAC-based model, but is a pure NAR model (+ autoregressive length task) .
 	+ Originally thought to be bunk from inferencing tests having audio drastically drop off into silence, but I suppose it was just some issue that eventually resolved itself.
+      + Addendum: I don't know what magic I did for that model, but I cannot recreate a decent EnCodec-backed model instead, despite the test trainer working fine.
 	+ Suffers from the same problems the above model suffers from (terrible quality).
 	+ *Huge* performance gains, but may definitely suffer from some specific qualities in the outputs, if it does get trained right.
 
@@ -81,9 +82,13 @@ Under `./models/experiments/` are some failed models, but are included to serve
 	+ The model definitely needs to be retrained as there's some errors for the additional tokens.
 		+ If these cannot be nailed out with more training, then I imagine a similar approach to speculative decoding where the nth tokens are discarded if the confidence is low.
 		+ Greedy sampling might be beneficial instead for this, as the NAR does benefit greatly from low temperatures / greedy sampling.
+    + It seems naively just adjusting the "causal size" (amount of tokens to predict into the future, and in turn, how many tokens are returned per step) introduces crackles at fixed intervals.
 
 Some additional configurations have been explored with, but experiments have not been fruitful:
-
 * Exotic wrappers like `BitNet` seemed to yield little gains in inferencing, somehow. The memory savings is pretty much unneccessary as the models are already manageable at ~200M parameters.
+* Mamba / Mamba2-based models have shown that it's ***really*** hard to have an AR+NAR model. I really do not want to bother throwing the compute at another ~~meme~~ arch I can't easily make use of all the other tech to throw at.
 
-* Mamba / Mamba2-based models have shown that it's ***really*** hard to have an AR+NAR model. I really do not want to bother throwing the compute at another ~~meme~~ arch I can't easily make use of all the other tech to throw at.
+Some current "achitectural features" are in-use, but their effects need to be experimented with further:
+* `split_classifier_heads` is still a mystery whether it's truly helpful or not (each RVQ level gets its own output head).
+* `audio_embeddings_sum` is also a mystery whether it matters if each later RVQ level should "see" the past levels through summing embeddings, or if not doing it is preferable.
+* Disabling `unified_position_ids` seems to help quality more often than not, but I'm still unsure if it's beneficial in practice.