---
license: other
base_model: lewtun/gemma-7b-sft-full-deita-10k-v0
tags:
- alignment-handbook
- trl
- dpo
- generated_from_trainer
- trl
- dpo
- generated_from_trainer
datasets:
- HuggingFaceH4/ultrafeedback_binarized
- HuggingFaceH4/orca_dpo_pairs
model-index:
- name: gemma-7b-dpo-full-mix2-beta-0.1
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# gemma-7b-dpo-full-mix2-beta-0.1

This model is a fine-tuned version of [lewtun/gemma-7b-sft-full-deita-10k-v0](https://huggingface.co/lewtun/gemma-7b-sft-full-deita-10k-v0) on the HuggingFaceH4/ultrafeedback_binarized and the HuggingFaceH4/orca_dpo_pairs datasets.
It achieves the following results on the evaluation set:
- Loss: 0.4056
- Rewards/chosen: -0.3995
- Rewards/rejected: -3.5721
- Rewards/accuracies: 0.7926
- Rewards/margins: 3.1726
- Logps/rejected: -414.5198
- Logps/chosen: -392.3416
- Logits/rejected: 83.8425
- Logits/chosen: 83.1641

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 2
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6146        | 0.18  | 100  | 0.4651          | -2.3366        | -4.2121          | 0.7527             | 1.8755          | -420.9197      | -411.7124    | 84.3991         | 81.8795       |
| 0.5464        | 0.35  | 200  | 0.4531          | -0.7850        | -3.1857          | 0.7899             | 2.4007          | -410.6562      | -396.1968    | 84.8764         | 82.9057       |
| 0.5841        | 0.53  | 300  | 0.4209          | -1.5926        | -4.2403          | 0.8085             | 2.6477          | -421.2023      | -404.2725    | 83.7612         | 81.9224       |
| 0.519         | 0.7   | 400  | 0.4162          | -1.2384        | -4.1774          | 0.7819             | 2.9390          | -420.5732      | -400.7308    | 85.8201         | 84.7816       |
| 0.5432        | 0.88  | 500  | 0.4134          | -0.3763        | -3.5060          | 0.8032             | 3.1296          | -413.8586      | -392.1099    | 83.6363         | 82.8991       |


### Framework versions

- Transformers 4.39.0.dev0
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.15.1