---
base_model: meta-llama/Meta-Llama-3.1-70B-Instruct
library_name: peft
---

# Model Card for Model ID

This LoRA adapter was extracted from
[mlabonne/Meta-Llama-3.1-70B-Instruct-lorablated](https://huggingface.co/mlabonne/Meta-Llama-3.1-70B-Instruct-lorablated)
and uses
[meta-llama/Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct)
as a base.

## Model Details

The model was extracted by running
[mlabonne/harmful_behaviors](https://huggingface.co/datasets/mlabonne/harmful_behaviors)
and the user prompts (but not assistant responses or system messages) from
[Guilherme34/uncensor](https://huggingface.co/datasets/Guilherme34/uncensor)
through the original abliterated model to generate a dataset of
prompt/completion pairs, and was trained for 2 epochs on a 8xA100s with Axolotl
using FSDP. Since the original abliterated model isn't perfect at avoiding
refusals, the dataset was cleaned to remove the few refusals generated prior to
training.

### Model Description

- **Developed by:** @reissbaker
- **Funded by:** Synthetic Lab
- **License:** Apache 2.0
- **Finetuned from model:** Llama 3.1 70B Instruct

## How to Get Started with the Model

Run the model with one click on [glhf.chat](https://glhf.chat).


#### Training Hyperparameters

* BF16 mixed-precision
* 4e-4 LR
* Linear LR schedule
* Fused AdamW optimizer