No matter what I do the model only outputs 'garbage' text.
I'm by no means trying to insult the model, but I've used 100s of models and never saw this problem, the only output I can get out of it is something like this.
finira अ sontकature letותengthever但 >_sok）lex Jenaùёetteيلode(n Gu قigenALfe su going title thenา h а number зamesrect Рquest？” والferory)Vunc headusatherutezi componrefроaveivo patients
I have it on Mistral settings and tried others, but it simply just won't do anything. I AM using the GGUF version from Mrader however. Literally no one talks about this model so I just decided to post here because I really like the visual novels in the description and wanted to test this.
What am I doing wrong ? :(
I'm using MarinaraSpaghetti Customized/Mistral Improved context+instruct preset with Spiratrioth's sphiratrioth666/SillyTavern-Presets-Sphiratrioth system prompt+samplers its working fine for me but it always keep overwriting. Also i'm on V2.0 GGUF
Not sure if the Mrader model is "V2 GGUF". Good to know about the presets, hopefully the model maker can chime it for optimal presets and update the model main page. I'll try Mari's, didn't think I'd need anything past basic settings.
Hello there, Thank you so much for your interest and reaching out!
I'm not familiar with GGUF or SillyTavern at the moment, as I'm currently using vLLM for testing and inference.
However, I'd be happy to share the inference settings I've been using with vLLM, as they might be helpful for your use case.
Here are the parameters I’m using:
Temperature: 0.5 to 0.9
Min P: 0.05 to 0.1
Repetition Penalty: 1.05 to 1.1
Stop Words: [/INST], <|im_end|>,
I hope this helps! Let me know if you have any other questions.