Lewdiculous

/

Model-Requests

like 37

Performance

We evaluate our models on Open LLM Leaderboard, they achieve good performance.

Usage

The chat template of our chat models is similar as Official gemma-7b-it:

<bos><start_of_turn>user
hello, who are you?<end_of_turn>
<start_of_turn>model
I am a AI program developed by Firefly<eos>

You can use script to inference in Firefly.

You can also use the following code:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name_or_path = "YeungNLP/firefly-gemma-7b"
model = AutoModelForCausalLM.from_pretrained(
    model_name_or_path,
    trust_remote_code=True,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16,
    device_map='auto',
)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)

prompt = "Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions. "
text = f"""
<bos><start_of_turn>user
{prompt}<end_of_turn>
<start_of_turn>model
""".strip()
model_inputs = tokenizer([text], return_tensors="pt").to('cuda')

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=1500,
    top_p = 0.9,
    temperature = 0.35,
    repetition_penalty = 1.0,
    eos_token_id=tokenizer.encode('<eos>', add_special_tokens=False)
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

IQ2 Series Q3_K-M(is it's performance better than IQ3_M? idk.)

You can refer to these for a general overview of the current sizes.

@Cran-May Can you add what would be intended use cases/applications for this particular model?

(I am thinking general simple/summary assistant work?)

Just because it doesn't seem like what I usually focus on -- un-aligned and unsafe models with few restrictions.

Cropped and upscaled card image: (square, 1:1)

    quantization_options = [
        "IQ2_XXS", "IQ2_XS", "IQ2_S", "IQ2_M", "Q3_K_M",
        "Q4_K_M", "Q4_K_S", "IQ4_XS", "Q5_K_M", "Q5_K_S",
        "Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"
    ]

IQ2's will take a good while longer.

@Cran-May - Heya! Will wait for your inputs.

I will need assistance to continue in this case as I am unable to load the initial FP16 GGUF created from your repo:

llama_model_load: error loading model: create_tensor: tensor 'blk.0.attn_q.weight' has wrong shape; expected  3072,  3072, got  3072,  4096,     1,     1
llama_load_model_from_file: failed to load model
main: error: unable to load model

Can't say I ever ran a gemma model.

@Virt-io Just in case you haven't seen this but do you know if I can sort this on my own?

I can't seem to load the FP16 GGUF converted from the linked repo using llama.cpp/convert.py. Maybe you know something about Gemma and Llama.cpp I am not aware...

@Lewdiculous

Use convert-hf-to-gguf.py instead of convert.py

I am unsure why convert.py doesn't work as I have not been paying attention to gemma models.

Warning convert-hf-to-gguf.py gave me a FP32 gguf so you will need to requant it to FP16 before running imatrix. In addition convert-hf-to-gguf.py needs more ram, or a page file.

convert-hf-to-gguf.py gave me a FP32 gguf

Love it xD

I'll try ty! I've never touched a gemma model :')

Maybe I should just be using convert-hf-to-gguf.py all along.

I consider 30/32GB RAM usage optimal. LOL. But yea, pagefile is there.

I think this is what they call a Skill Issue. Thank you, Virt.

@Cran-May - Things seem fine, quanting, I'd really like to see your tests for the quants as I'm curious about these Gemma IQ2-imatrix-quants.

Will soon be available at:
https://huggingface.co/Lewdiculous/firefly-gemma-7b-GGUF-IQ-Imatrix