What dataset is used for fine-tuning?
Hello Maziyar,
Thank you for uploading this awesome mini model. Can you please let us know if you used a custom dataset or existing HF dataset to fine-tune this model? How many examples/tokens of training dataset did you use? Also, can you please share if you did full fine-tuning with 8xA100 or LORA fine-tuning with a single GPU? I want to train my own mini model to write in a new language (Uzbek), but I am struggling to find any details on what kind of training I should do (e.g. should I do text completion on all 350k uzbek wiki articles first and then do LORA fine-tuning with chat examples - 100k chats - I translated into Uzbek with NLLB?). I tried LORA with r=256 and a=512 and different combinations with LLAMA3 8B instruct, Mistral 7b v0.3 instruct but models are still not proficient with the grammar and cannot transfer knowledge from English to Uzbek.
Salomat boshet/ Thank you!
Hi @MaziyarPanahi , please, let me know when you get a chance. Thanks!