Hugging Face's logo

VAGOsolutions
/
Llama-3-SauerkrautLM-8b-Instruct

Text Generation
Transformers
Safetensors
German
English
llama
two stage dpo
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
10

Context window

#4
by tapos999 - opened
Discussion
tapos999

Hi, really nice fine-tune specially working with German language usecases. Any plan to extend context window or something similar like llama3-gradient?

Thanks

DavidGF
 VAGO solutions org

Hey @tapos999
you can change the model config.json:

from 

rope_scaling": null,

to

  "rope_scaling": {
     "type": "dynamic",
     "factor": 4.0
  },

and you get a 32K context size with no performance decrease!

HF,
David

DavidGF changed discussion status to closed
tapos999

Hi @DavidGF thanks for responding. One more question, I am deploying it via vllm, I see there's also a rope_theta which is None. would you recommend any specific rope_theta value with this factor 4.0?

Thanks again

Sign up or log in to comment