README.md

---
base_model:
- google/flan-t5-small
- google/flan-t5-large
- google/flan-t5-xl
---

## 🧠 Flan-T5-{Small|Large|XL}-RPO

> 🔬 Fine-tuned with **Reward Partitioning Optimization (RPO)** — a value-free, stable method for single-trajectory reinforcement learning with scalar feedback.

---

### 📌 Model Summary

This model is a fine-tuned variant of the [Flan-T5](https://huggingface.co/google/flan-t5) {Small|Large|XL} checkpoint, trained using **Reward Partitioning Optimization (RPO)**. RPO is a new method designed for learning from single-trajectory scalar feedback (e.g., thumbs up/down), and eliminates the need for learning value functions or preference pairs.

* ✅ Trained with only (prompt, response, reward) triplets.
* 🔁 No joint optimization, no auxiliary models.
* 🚀 Efficient and stable training.
* 🤖 Strong preference alignment (evaluated by LLM-as-a-judge).
* 📊 Outperforms KTO and DRO in automatic metrics and LLM preference winrate.

---

### 🧪 Training Details

* **Base Model:** `flan-t5-{small|large|xl}`
* **Dataset:** [UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) — high-quality (prompt, response, reward) triplets with multiple completions per prompt.
* **Feedback Format:** scalar reward (e.g., \[prompt, response, reward]).
* **GPU Used:** 1× A100 (80GB)
* **Training Objective:** RPO supervised learning using partitioned reward normalization.
* **Baselines Compared:** DRO and KTO.

---

### 🤖 Inference

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_name = "bilalfaye/flan-t5-{small|large|xl}-rpo"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)

prompt = "How can I improve my productivity working from home?"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.2,
    no_repeat_ngram_size=3,
)

response = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
print(response)
```

---

### 📈 Evaluation Summary

| Judge   | Win Rate vs DRO | Win Rate vs KTO | Win Rate vs SFT |
| ------- | --------------- | --------------- | --------------- |
| Mistral | ✅ **83–93%**    | ✅ **82–93%**    | ✅ **82–84%**    |
| LLaMA   | ✅ **67–74%**    | ✅ **65–72%**    | ✅ **63–73%**    |

---

### ✅ Use Cases

* Aligned conversational agents
* Helpful, non-toxic instruction following
* Scalar feedback training pipelines
* Preference-optimized generation (without pairwise preference labels)

---

### 📚 Citation

If you use this model, please cite the following paper:

```bibtex
@article{faye2025rpo,
  title     = {Value-Free Policy Optimization via Reward Partitioning},
  author    = {Bilal Faye and Hanane Azzag and Mustapha Lebbah},
  journal   = {arXiv preprint arXiv:2406.XXXX},
  year      = {2025}
}
```

---

### 🔗 Related Models

* `bilalfaye/flan-t5-small-rpo`
* `bilalfaye/flan-t5-large-rpo`
* `bilalfaye/flan-t5-xl-rpo`