---
license: apache-2.0
base_model: TheBloke/OpenHermes-2-Mistral-7B-GPTQ
tags:
- generated_from_trainer
model-index:
- name: openhermes-mistral-dpo-gptq
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# openhermes-mistral-dpo-gptq

This model is a fine-tuned version of [TheBloke/OpenHermes-2-Mistral-7B-GPTQ](https://huggingface.co/TheBloke/OpenHermes-2-Mistral-7B-GPTQ) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 10.4812
- Rewards/chosen: -34.3483
- Rewards/rejected: -10.5927
- Rewards/accuracies: 0.4375
- Rewards/margins: -23.7556
- Logps/rejected: -369.0945
- Logps/chosen: -652.6519
- Logits/rejected: -1.9434
- Logits/chosen: -2.1850

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2
- training_steps: 50

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.785         | 0.01  | 10   | 0.6767          | 0.0342         | 0.0037           | 0.5625             | 0.0305          | -263.1300      | -308.8264    | -2.4042         | -2.7773       |
| 0.988         | 0.01  | 20   | 0.6634          | 0.1443         | -0.0047          | 0.75               | 0.1490          | -263.2147      | -307.7260    | -2.4189         | -2.7870       |
| 0.9481        | 0.01  | 30   | 10.5125         | -35.0203       | -10.9252         | 0.4375             | -24.0951        | -372.4191      | -659.3712    | -1.9468         | -2.1781       |
| 10.47         | 0.02  | 40   | 17.8422         | -55.0969       | -26.2159         | 0.375              | -28.8810        | -525.3258      | -860.1373    | -1.6618         | -1.8474       |
| 2.5456        | 0.03  | 50   | 10.4812         | -34.3483       | -10.5927         | 0.4375             | -23.7556        | -369.0945      | -652.6519    | -1.9434         | -2.1850       |


### Framework versions

- Transformers 4.34.1
- Pytorch 2.0.1+cu117
- Datasets 2.14.6
- Tokenizers 0.14.1