BigHuggyD/BigHuggyD-Grey-WizardLM-2-8x22B

Grey-WizardLM-2-8x22B

Just a little experiment with the Model Stock method of merging. This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the Model Stock merge method using alpindale/WizardLM-2-8x22B as a base.

Models Merged

The following models were included in the merge:

• lightblue/Karasu-Mixtral-8x22B-v0.1
• openbmb/Eurux-8x22b-nca
• cognitivecomputations/dolphin-2.9.2-mixtral-8x22b
• migtissera/Tess-2.0-Mixtral-8x22B
• fireworks-ai/mixtral-8x22b-instruct-oh

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: alpindale/WizardLM-2-8x22B
  - model: lightblue/Karasu-Mixtral-8x22B-v0.1
  - model: fireworks-ai/mixtral-8x22b-instruct-oh
  - model: cognitivecomputations/dolphin-2.9.2-mixtral-8x22b
  - model: openbmb/Eurux-8x22b-nca
  - model: migtissera/Tess-2.0-Mixtral-8x22B
base_model: alpindale/WizardLM-2-8x22B
merge_method: model_stock
dtype: bfloat16