❗⚠️ WARNING ⚠️❗
❗ This model has been deprecated due to a sliding window error in the base model's configuration. This issue has been resolved with the following commit in the base model, and upcoming versions of the Einstein series will utilize the correct configuration in the base model.
🔬 Einstein-v5-v0.2-7B
This model is a full fine-tuned version of alpindale/Mistral-7B-v0.2-hf on diverse datasets.
This model is finetuned using 8xRTX3090
+ 1xRTXA6000
using axolotl.
This model's training was sponsored by sablo.ai.
See axolotl config
axolotl version: 0.4.0
base_model: alpindale/Mistral-7B-v0.2-hf
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true
load_in_8bit: false
load_in_4bit: false
strict: false
chat_template: chatml
datasets:
- path: data/merged_all.json
ds_type: json
type: alpaca
conversation: chatml
- path: data/gpteacher-instruct-special-alpaca.json
ds_type: json
type: gpteacher
conversation: chatml
- path: data/capybara_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/synthia-v1.3_sharegpt_12500.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/slimorca_dedup_filtered_95k_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json
ds_type: json
type: sharegpt
strict: false
conversation: chatml
- path: data/pippa_bagel_repo_3k_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/gpt4_data_lmys_1m_sharegpt.json
ds_type: json
type: sharegpt
conversation: chatml
- path: data/sharegpt_gpt4_english.json
ds_type: json
type: sharegpt
conversation: chatml
dataset_prepared_path: last_run_prepared
# val_set_size: 0.005
val_set_size: 0.0
do_bench_eval: true
output_dir: ./Einstein-v5-Mistral-v0.2-beta-model
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false
wandb_project: Einstein
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
hub_model_id: Weyaxi/Einstein-v5-Mistral-v0.2-beta
save_safetensors: true
gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 2
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
evals_per_epoch: 3 # changed
eval_table_size:
eval_table_max_new_tokens: 128
saves_per_epoch: 3 # changed
debug:
deepspeed: zero3_bf16.json
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: "<s>"
eos_token: "<|im_end|>"
unk_token: "<unk>"
tokens:
- "<|im_start|>"
💬 Prompt Template
You can use this prompt template while using the model:
ChatML
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>
This prompt template is available as a chat template, which means you can format messages using the
tokenizer.apply_chat_template()
method:
messages = [
{"role": "system", "content": "You are helpful AI asistant."},
{"role": "user", "content": "Hello!"}
]
gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
model.generate(**gen_input)
🔄 Quantizationed versions
Quantizationed versions of this model is available.
GGUF @bartowski
ExLlamaV2 @bartowski
🎯 Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 65.65 |
AI2 Reasoning Challenge (25-Shot) | 60.92 |
HellaSwag (10-Shot) | 80.99 |
MMLU (5-Shot) | 61.02 |
TruthfulQA (0-shot) | 52.59 |
Winogrande (5-shot) | 78.69 |
GSM8k (5-shot) | 59.67 |
🤖 Additional information about training
This model is full fine-tuned for 1 epoch.
Total number of steps was 1124.
🤝 Acknowledgments
Thanks to sablo.ai for sponsoring this model.
Thanks to all the dataset authors mentioned in the datasets section.
Thanks to axolotl for making the repository I used to make this model.
Thanks to all open source AI community.
If you would like to support me:
- Downloads last month
- 12
Model tree for Weyaxi/Einstein-v5-v0.2-7B
Base model
mistral-community/Mistral-7B-v0.2Datasets used to train Weyaxi/Einstein-v5-v0.2-7B
Collection including Weyaxi/Einstein-v5-v0.2-7B
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard60.920
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard80.990
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard61.020
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard52.590
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard78.690
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard59.670