--- license: llama2 language: - hu - en tags: - puli - llama - finetuned base_model: ariel-ml/PULI-LlumiX-32K-instruct-f16-0.2 pipeline_tag: text-generation --- # PULI LlumiX 32K instruct (6.74B billion parameter) Intruct finetuned version of NYTK/PULI-LlumiX-32K. ## Provided files | Quant method | Bits | Use case | | ---- | ---- | ---- | | Q3_K_M | 3 | very small, high quality loss | | Q4_K_S | 4 | small, greater quality loss | | Q4_K_M | 4 | medium, balanced quality - recommended | | Q5_K_S | 5 | large, low quality loss - recommended | | Q5_K_M | 5 | large, very low quality loss - recommended | | Q6_K | 6 | very large, extremely low quality loss | | Q8_0 | 8 | very large, extremely low quality loss - not recommended | ## Training platform [Runpod](https://runpod.ui) RTX 4090 GPU ## Hyper parameters - Epoch: 3 - LoRA rank (r): 16 - LoRA alpha: 16 - Lr: 2e-4 - Lr scheduler: cosine - Optimizer: adamw_8bit - Weight decay: 0.01 ## Dataset boapps/szurkemarha Only Hungarian instructions were selected: ~53000 prompts. ## Prompt format: ChatML ``` <|im_start|>system Egy segítőkész mesterséges intelligencia asszisztens vagy. Válaszold meg a kérdést legjobb tudásod szerint!<|im_end|> <|im_start|>user Ki a legerősebb szuperhős?<|im_end|> <|im_start|>assistant A legerősebb szuperhős a Marvel univerzumában Hulk.<|im_end|> ``` ## Base model - Trained with OpenChatKit [github](https://github.com/togethercomputer/OpenChatKit) - The [LLaMA-2-7B-32K](https://huggingface.co/togethercomputer/LLaMA-2-7B-32K) model were continuously pretrained on Hungarian dataset - The model has been extended to a context length of 32K with position interpolation - Checkpoint: 100 000 steps ## Base model dataset for continued pretraining - Hungarian: 7.9 billion words, documents (763K) that exceed 5000 words in length - English: Long Context QA (2 billion words), BookSum (78 million words) ## Limitations - max_seq_length = 32 768 - float16 - vocab size: 32 000