|
|
--- |
|
|
license: cc-by-nc-4.0 |
|
|
datasets: |
|
|
- jinzhuoran/OmniRewardData |
|
|
base_model: |
|
|
- openbmb/MiniCPM-o-2_6 |
|
|
--- |
|
|
|
|
|
|
|
|
|
|
|
# Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences |
|
|
|
|
|
|
|
|
<p align="center"> |
|
|
<a href="https://huggingface.co/datasets/HongbangYuan/OmniRewardBench"> 🤗 Benchmark</a></a> | |
|
|
<a href="https://hf.co/datasets/jinzhuoran/OmniRewardData"> 🤗 Dataset</a> | |
|
|
<a href="https://hf.co/jinzhuoran/OmniRewardModel"> 🤗 Model</a> | |
|
|
<a href="https://omnireward.github.io/"> 🏠 Homepage</a> |
|
|
</p> |
|
|
|
|
|
|
|
|
|
|
|
## 🧩 Overview |
|
|
|
|
|
**OmniRewardModel** is our pretrained **discriminative reward model** designed to handle *omni-modal* tasks (e.g., text, image, video) and *free-form human preferences*. |
|
|
|
|
|
It is built upon the open-source base model [MiniCPM-o-2_6](https://huggingface.co/openbmb/MiniCPM-o-2_6), with an additional **value head** appended to produce scalar reward scores. |
|
|
|
|
|
The model supports fine-grained scoring across various tasks and modalities, and can be seamlessly loaded via Hugging Face Hub. |
|
|
|
|
|
--- |
|
|
|
|
|
|
|
|
## 🛠️ Environment Setup |
|
|
|
|
|
|
|
|
To reproduce the training process in our paper, please make sure to set up the environment as described below. |
|
|
Our training code is built upon the [llama-factory](https://github.com/hiyouga/llama-factory) framework. |
|
|
|
|
|
```bash |
|
|
git clone https://github.com/HongbangYuan/OmniReward.git |
|
|
conda create -n omnireward python=3.10 |
|
|
conda activate omnireward |
|
|
``` |
|
|
|
|
|
We recommend using **`torch==2.2.0`** for best compatibility. |
|
|
|
|
|
Install PyTorch (choose one based on your CUDA version): |
|
|
|
|
|
```bash |
|
|
# For CUDA 11.8: |
|
|
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 \ |
|
|
--index-url https://download.pytorch.org/whl/cu118 |
|
|
|
|
|
# For CUDA 12.1: |
|
|
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 \ |
|
|
--index-url https://download.pytorch.org/whl/cu121 |
|
|
``` |
|
|
|
|
|
Then install the remaining dependencies: |
|
|
|
|
|
```bash |
|
|
cd OmniReward/OmniReward-Factory |
|
|
pip install -r requirements.txt |
|
|
``` |
|
|
|
|
|
## 📦 Data Preparation |
|
|
|
|
|
Download all required training and evaluation datasets from [OmniRewardData](https://huggingface.co/datasets/jinzhuoran/OmniRewardData) and [OmniRewardBench](https://huggingface.co/datasets/HongbangYuan/OmniRewardBench): |
|
|
|
|
|
```bash |
|
|
cd OmniReward-Factory |
|
|
bash scripts/download.sh |
|
|
``` |
|
|
|
|
|
## 🏋️♀️ Training Omni-Reward |
|
|
|
|
|
To reproduce the training results described in our paper, please navigate to the OmniReward-Factory directory and run the following scripts: |
|
|
|
|
|
```bash |
|
|
cd OmniReward-Factory |
|
|
bash scripts/train.sh |
|
|
bash scripts/train_t2t.sh |
|
|
bash scripts/train_ti2t.sh |
|
|
bash scripts/train_t2iv.sh |
|
|
``` |
|
|
## 📈 Loading and Evaluating Omni-Reward |
|
|
|
|
|
You can also directly use our pretrained Omni-Reward for evaluation without retraining. |
|
|
|
|
|
The models are publicly available at: |
|
|
|
|
|
👉 https://huggingface.co/jinzhuoran/OmniRewardModel |
|
|
|
|
|
```bash |
|
|
cd OmniReward-Factory |
|
|
bash scripts/eval_t2t.sh |
|
|
bash scripts/eval_t2t_tie.sh |
|
|
bash scripts/eval_ti2t.sh |
|
|
bash scripts/eval_ti2t_tie.sh |
|
|
``` |
|
|
|
|
|
- `--eval_dataset`: Specifies the evaluation dataset (e.g., `omni_t2t`, `omni_t2i`, `omni_t2v`, etc.). |
|
|
|
|
|
- `--eval_tie`: Enables w/ Ties evaluation. |
|
|
|
|
|
|
