🔬 Einstein-7B
This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on datasets related to science.
This model is fine-tuned using QLoRa and axolotl.
This model's training was sponsored by sablo.ai.
See axolotl config
axolotl version: 0.3.0
base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: sci-datasets/arc_challange_train_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/camelai_biology_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/camelai_chemistry_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/camelai_physics_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/openbookqa_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/reclor_science_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/scibench_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/scienceqa_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/theoremqa_alpaca.json
ds_type: json
type: alpaca
- path: sci-datasets/tiger_scienceeval_alpaca.json
ds_type: json
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0
output_dir: ./science-mistral
adapter: qlora
lora_model_dir:
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
lora_r: 128
lora_alpha: 64
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
wandb_project: huggingface
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
hub_model_id: Weyaxi/science-mistral
# change #
gradient_accumulation_steps: 12
micro_batch_size: 6
num_epochs: 2
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
# change #
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
saves_per_epoch: 3
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
bos_token: "<s>"
eos_token: "</s>"
unk_token: "<unk>"
📊 Datasets
Following datasets were used in this model:
ARC (Note: Only train part)
💬 Prompt Template
You can use this prompt template while using the model:
Alpaca
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Input:
{input}
### Response:
🤝 Acknowledgments
Thanks to Platypus for providing scripts to convert some of the datasets to Alpaca format: Platypus/data_pipeline
Thanks to all the dataset authors mentioned in the datasets section.
Thanks to axolotl for making the repository I used to make this model.
If you would like to support me:
- Downloads last month
- 22
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for Weyaxi/Einstein-7B
Base model
mistralai/Mistral-7B-v0.1
Finetuned
this model
Datasets used to train Weyaxi/Einstein-7B
Collection including Weyaxi/Einstein-7B
Collection
Einstein series
•
11 items
•
Updated
•
6