Chatty-McChatterson-3-mini-128k
Model Details
Model Name: Chatty-McChatterson-3-mini-128k
Base Model: microsoft/Phi-3-mini-128k-instruct
Fine-tuning Method: Supervised Fine-Tuning (SFT)
Dataset: ultrachat_200k
Training Data: 12884 conversations selected for being 512 input tokens or less
Training Duration: 4 hours
Hardware: Nvidia RTX A4500
Epochs: 3
Training Procedure
This model was fine-tuned to provide better instructions on code.
The training was conducted using PEFT and SFTTrainer on select conversations from the Ultra Chat 200k dataset. Training was completed in 3 epochs (19326 steps) over a span of 4 hours on an Nvidia A4500 GPU.
The dataset comprised of a filterd list of rows from the Ultra Chat 200k dataset, where the prompt template was 512 tokens or less.
Intended Use
This model is designed to improve the overall chat experience and response quality.
Getting Started
Instruct Template
<|system|>
{system_message} <|end|>
<|user|>
{Prompt) <|end|>
<|assistant|>
Transfromers
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_name_or_path = "thesven/Chatty-McChatterson-3-mini-128k"
# BitsAndBytesConfig for loading the model in 4-bit precision
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="float16",
)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
model_name_or_path,
device_map="auto",
trust_remote_code=False,
revision="main",
quantization_config=bnb_config
)
model.pad_token = model.config.eos_token_id
prompt_template = '''
<|user|>
What is the name of the big tower in Toronto?.<|end|>
<|assistant|>
'''
input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.1, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=256)
generated_text = tokenizer.decode(output[0, len(input_ids[0]):], skip_special_tokens=True)
print(generated_text)
- Downloads last month
- 5