metadata

base_model: unsloth/gemma-2-9b-bnb-4bit
language:
  - en
license: apache-2.0
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - gemma2
  - gguf

Uploaded model

Developed by: Deeokay
License: apache-2.0
Finetuned from model : unsloth/gemma-2-9b-bnb-4bit

This gemma2 model was trained 2x faster with Unsloth and Huggingface's TRL library.

README

This is a test model on a the following

a private dataset
slight customization on alpaca chat template
Works with Ollama create but requires customization to Modelfile
One reason for this was wanted to try doing Q2_K and see if it was actually good(?)
My examples will be based on unslot.Q2_K.GGUF file, however other quantization should work as well

HOW TO USE

The whole point of conversion for me was I wanted to be able to to use it through Ollama or (other local options) For Ollama, it required to be a GGUF file. Once you have this it is pretty straight forward

Quick Start:

You must already have Ollama running in your setting
Download the unsloth.Q4_K_M.gguf model from Files
In the same directory create a file call "Modelfile"
Inside the "Modelfile" type

FROM ./GEMMA2_unsloth.Q2_K.gguf

TEMPLATE """Below are instructions that describe the task. Write response that appropriately complete the request.
{{ if .Prompt }}<start_of_turn>
### Instruction:{{ .Prompt }}<eos>
{{ end }}<start_of_turn>
### Response:<eos>"""

PARAMETER stop "<start_of_turn>"
PARAMETER stop "<end_of_turn>"
PARAMETER stop "<eos>"
PARAMETER stop "### "
PARAMETER stop "###: "
PARAMETER stop "###"

PARAMETER temperature 0.4

SYSTEM You are a teacher for a young curious mind. Please be accurate and percise. Do not make up facts, you must be base your response on data and facts.

Save a go back to the folder (folder where model + Modelfile exisit)
Now in terminal make sure you are in the same location of the folder and type in the following command

ollama create mycustomai  # "mycustomai" <- you can name it anything u want

After than you should be able to use this model to chat! This GGUF is based on Gemma2-9B-4bit by Unslot,

NOTE: DISCLAIMER

Please note this is not for the purpose of production, but result of Fine Tuning through self learning

This is my first pass through my ~30K personalized customized dataset.

Please feel free to customize the Modelfile, and if you do get a better response than mine, please share!!

If would like to know how I started creating my dataset, you can check this link Crafting GPT2 for Personalized AI-Preparing Data the Long Way (Part1)

As the data was getting created with custom GPT2 special tokens, I had to convert that to the a Alpaca Template.

However I got creative again.. the training data has the following Template:

prompt = """Below is an instruction that describes a task, with an analysis that provides further context. Write a response, classification and sentiment that appropriately completes the request.

### Instruction:
{}

### Analysis:
{}

### Response:
{}

### Classification:
{}

### Sentiment:
{}
"""
EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN

def formatting_prompts_func(examples):
    instructions    = examples["Question"]
    analyses        = examples["Analysis"]
    responses       = examples["Answer"]
    classifications = examples['Classification']
    sentiments      = examples['Sentiment']
    texts = []
    for instruction, analysis, response, classification, sentiment in zip(instructions, analyses, responses, classifications, sentiments):
        # Must add EOS_TOKEN, otherwise your generation will go on forever!
        text = prompt.format(instruction, analysis, response, classification, sentiment) + EOS_TOKEN
        texts.append(text)
    return { "text" : texts }

data_path = 'file to dataset'
df = pd.read_csv(data_path)
dataset = Dataset.from_pandas(df)

# Shuffle the dataset
shuffled_dataset = dataset.shuffle(seed=42)  # Seed for reproducibility

dataset = dataset.map(formatting_prompts_func, batched = True,)

Will be updating this periodically.. as I have limited colab resources..