--- license: apache-2.0 library_name: peft tags: - finetuned - multimodal base_model: mistralai/Mixtral-8x7B-Instruct-v0.1 dataset: ./out inference: false --- These are weights for a version of `mistralai/Mixtral-8x7B-Instruct-v0.1` finetuned for multimodal applications. ### Modalities * CLIPVisionModality (use `` in text and provide `images`, encoded as 576 tokens) ### Usage GitHub: https://github.com/sshh12/multi_token (includes training scripts and basic inference server) ### Dataset ./out (558128 examples) ``` {'id': '004539375', 'images': ['/data/llava_pretrain_data/images/00453/004539375.jpg'], 'messages': [{'content': 'Render a clear and concise summary of the photo.\n', 'role': 'user'}, {'content': 'select luxury furniture 3 - inch gel memory foam mattress topper', 'role': 'assistant'}]} ``` ### Training Device(s) ``` name, pci.bus_id, vbios_version NVIDIA A100 80GB PCIe, 00000000:61:00.0, 92.00.90.00.0F ``` ### Model ``` MistralLMMForCausalLM.model = PeftModelForCausalLM( (base_model): LoraModel( (model): MistralLMMForCausalLM( (model): MistralLMMModel( (embed_tokens): Embedding(32000, 4096) (layers): ModuleList( (0-31): 32 x MistralDecoderLayer( (self_attn): MistralAttention( (q_proj): lora.Linear( (base_layer): Linear(in_features=4096, out_features=4096, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=4096, out_features=64, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=64, out_features=4096, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (k_proj): lora.Linear( (base_layer): Linear(in_features=4096, out_features=1024, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=4096, out_features=64, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=64, out_features=1024, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (v_proj): lora.Linear( (base_layer): Linear(in_features=4096, out_features=1024, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=4096, out_features=64, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=64, out_features=1024, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (o_proj): lora.Linear( (base_layer): Linear(in_features=4096, out_features=4096, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=4096, out_features=64, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=64, out_features=4096, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (rotary_emb): MistralRotaryEmbedding() ) (mlp): MistralMLP( (gate_proj): lora.Linear( (base_layer): Linear(in_features=4096, out_features=14336, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=4096, out_features=64, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=64, out_features=14336, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (up_proj): lora.Linear( (base_layer): Linear(in_features=4096, out_features=14336, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=4096, out_features=64, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=64, out_features=14336, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (down_proj): lora.Linear( (base_layer): Linear(in_features=14336, out_features=4096, bias=False) (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=14336, out_features=64, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=64, out_features=4096, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (act_fn): SiLU() ) (input_layernorm): MistralRMSNorm() (post_attention_layernorm): MistralRMSNorm() ) ) (norm): MistralRMSNorm() (vision_clip_lmm_projector): Sequential( (0): Linear(in_features=1024, out_features=4096, bias=True) (1): GELU(approximate='none') (2): Linear(in_features=4096, out_features=4096, bias=True) ) ) (lm_head): Linear(in_features=4096, out_features=32000, bias=False) ) ) ) ``` ### Framework versions - PEFT 0.10.0