Upload README.md

README.md

@@ -1,199 +1,184 @@
 ---
-library_name: transformers
-tags: []
+license: apache-2.0
 ---
+# EurusPRM-Stage2
 
-# Model Card for Model ID
+## Links
 
-<!-- Provide a quick summary of what the model is/does. -->
+- 📜 [Blog]()
+- 🤗 [PRIME Collection](https://huggingface.co/PRIME-RL)
+- 🤗 [Training Data](https://huggingface.co/datasets/PRIME-RL/EurusPRM-Stage1-Data)
 
+## Introduction
 
+EurusPRM-Stage1 is trained using **[Implicit PRM](https://arxiv.org/abs/2412.01981)**, which obtains free process rewards at no additional cost but just needs to simply train an ORM on the cheaper response-level labels. During inference, implicit process rewards are obtained by forward passing and calculating the log-likelihood ratio on each step. It serves a great fundation for further training of **[EurusPRM-Stage2](https://huggingface.co/PRIME-RL/EurusPRM-Stage2)**.
 
-## Model Details
+<img src="./figs/implicit.png" alt="prm" style="zoom: 33%;" />
 
-### Model Description
+The key ingredient of Implicit PRM is the reward representation, as demonstrated below:
 
-<!-- Provide a longer summary of what this model is. -->
+<aside>
+✨
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+***Proposition***: Consider an ORM where the reward is parameterized by the log-likelihood ratio of two causal LMs, i.e.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+$$
+r_\phi(\mathbf{y}) := \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})}.
+$$
 
-### Model Sources [optional]
+Define
 
-<!-- Provide the basic links for the model. -->
+$$
+q_\phi^t(\mathbf{y}_{<t}, y_t) := \sum_{i=1}^{t} \beta \log \frac{\pi_\phi(y_{i}|\mathbf{y}_{<i})}{\pi_\text{ref}(y_{i}|\mathbf{y}_{<i})}.
+$$
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+is the exponential average of \\(r_\theta\\) at step \\(t\\).
 
-## Uses
+$$
+q_\phi^t(\mathbf{y}_{<t}, y_t) = \beta \log \mathbb{E}{\pi_\text{ref}(\mathbf{y}|\mathbf{y}_{\leq t})} \left[ e^{\frac{1}{\beta} r_\phi(\mathbf{y})} \right]
+$$
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+Hence, \\(q_\theta^t\\)represents an exact expectation of outcome reward \\(r_\theta\\) at step \\(t\\), i.e., the Q value.
 
-### Direct Use
+The proposition indicates that when modeling
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+$$
+r_\phi(\mathbf{y}) := \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})}
+$$
 
-[More Information Needed]
+to train an ORM with the standard pipeline, where \\(\beta\\) is a hyperparameter, \\(\phi\\) can implicitly learn a Q function. Hence, process reward \\(r_\phi^t\\) can be obtained by:
 
-### Downstream Use [optional]
+$$
+r_\phi^t := q_\phi^t - q_\phi^{t-1} = \beta \log \frac{\pi_\phi(y_{t}|\mathbf{y}_{<t})}{\pi_\text{ref}(y_{t}|\mathbf{y}_{<t})}.
+$$
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+Therefore, we can indeed obtain PRMs simply by collecting response-level data and training an ORM, without any burden of annotating step labels.
 
-[More Information Needed]
+The proposition is **agnostic to specific choices of the training objective of ORMs**. It can be instantiated with different objectives as vanilla ORM training, with the only difference being substituting the \\(r_\phi \left( \mathbf{y} \right)\\) with \\(\beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})}\\).
 
-### Out-of-Scope Use
+For example, DPO already meets our assumption and serves as a strong variant, while in this work, we instantiate our implicit PRM with cross entropy (CE) loss due to memory efficiency:
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+$$
+\small \mathcal{L}_{CE} = l \cdot \log \sigma \left( \beta \log \frac{\pi\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})} \right) + (1 - l) \cdot \log \left[ 1 - \sigma \left( \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})} \right) \right]
+$$
 
-[More Information Needed]
+We applied the above \\(L_{CE}\\) to train implicit PRM. We used a learning rate of 5e-7 and a batch-size of 64 for training. 
 
-## Bias, Risks, and Limitations
+## Usage
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+We show an example leveraging **EurusPRM-Stage2** below:
 
-[More Information Needed]
+```python
+coef=0.001
+d = {'query':'111','answer':['111','222']
+     }
+model = AutoModelForCausalLM.from_pretrained('PRIME-RL/EurusPRM-Stage2')
+tokenizer = AutoTokenizer.from_pretrained('PRIME-RL/EurusPRM-Stage2')
+ref_model = AutoModelForCausalLM.from_pretrained('Qwen/Qwen2.5-Math-7B-Instruct')
+input_ids = tokenizer.apply_chat_template([
+    {"role": "user", "content": d["query"]},
+    {"role": "assistant", "content": "\n\n".join(d["answer"])},
+], tokenize=True, add_generation_prompt=False)
+attention_mask = input_ids!=tokenizer.pad_token_id
+step_last_tokens = []
+for step_num in range(0, len(d["answer"])+1):
+    conv = tokenizer.apply_chat_template([
+        {"role":"user", "content":d["query"]},
+        {"role":"assistant", "content":"\n\n".join(d["answer"][:step_num])},
+    ], tokenize=False, add_generation_prompt=False)
+    conv = conv.strip()
+    if step_num!=0 and step_num!=len(d['answer']):
+        conv+='\n\n'
+    currect_ids = tokenizer.encode(conv,add_special_tokens=False)
+    step_last_tokens.append(len(currect_ids) - 2)
 
-### Recommendations
+inputs = {'input_ids':input_ids,'attention_mask':attention_mask,'labels':input_ids}
+step_last_tokens = torch.tensor(step_last_tokens)
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+def get_logps(model,inputs):
+    logits = model(input_ids=inputs['input_ids'], attention_mask=inputs['attention_mask']).logits
+    labels = inputs['labels'][:, 1:].clone().long()
+    logits = logits[:, :-1, :]
+    labels[labels == -100] = 0
+    per_token_logps = torch.gather(logits.log_softmax(-1), dim=2, index=labels.unsqueeze(2)).squeeze(2)
+    return per_token_logps
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+with torch.no_grad():
+    per_token_logps = get_logps(model, inputs)
+    per_token_logps = get_logps(ref_model,inputs)
 
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+raw_reward = per_token_logps - ref_per_token_logps
+beta_reward = coef * raw_reward
+beta_reward = beta_reward.cumsum(-1)
+beta_reward = beta_reward.gather(dim=-1, index=step_last_tokens[1:])
+print(beta_reward)
+```
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
+### Evaluation Base Model
 
-## Citation [optional]
+We adopt **Eurus-2-7B-SFT**, **Qwen2.5-7B-Instruct** and **Llama-3.1-70B-Instruct** as generation models to evaluate the performance of our implicit PRM. For all models, we set the sampling temperature as 0.5, *p* of the top-*p* sampling as 1.
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+### Best-of-N Sampling
 
-**BibTeX:**
+We use Best-of-64 as our evaluation metric. The weighting methods are different for several PRMs below.
 
-[More Information Needed]
+- For [Skywork-o1-Open-PRM-Qwen-2.5-7B](https://huggingface.co/Skywork/Skywork-o1-Open-PRM-Qwen-2.5-7B), we use simple average reward across all steps.
+- For EurusPRM-Stage 1, we use the minimum reward across all steps.
+- For EurusPRM-Stage 2, we use the accumulative rewards.
 
-**APA:**
+**Eurus-2-7B-SFT**
 
-[More Information Needed]
+| Method | Reward Model | MATH | AMC | AIME_2024 | OlympiadBench | Minerva Math | Avg |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| Greedy Pass @ 1 | N/A | 65.1 | 30.1 | 3.3 | 29.8 | 32.7 | 32.2 |
+| Majority Voting @ 64 | N/A | 65.6 | 53.0 | 13.3 | 39.1 | 22.4 | 38.7 |
+| Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 47.2 | 45.8 | 10.0 | 32.3 | 16.2 | 30.3 |
+|  | EurusPRM-Stage 1 | 44.6 | 41.0 | 6.7 | 32.9 | 17.3 | 28.5 |
+| Weighted Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 64.6 | **55.4** | 13.3 | **41.3** | 23.2 | 39.6 |
+|  | EurusPRM-Stage 1 | **66.0** | 54.2 | **13.3** | 39.6 | **29.0** | **40.4** |
 
-## Glossary [optional]
+**Llama-3.1-70B-Instruct**
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+| Method | Reward Model | MATH | AMC | AIME 2024 | OlympiadBench | Minerva Math | Avg |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| Greedy Pass @ 1 | N/A | 62.1 | 37.3 | 16.7 | 27.7 | 34.2 | 35.6 |
+| Majority Voting @ 64 | N/A | 80.2 | 53.0 | 26.7 | 40.4 | 38.6 | 47.8 |
+| Best-of-N @ 64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 77.8 | 56.6 | 23.3 | 39.0 | 31.6 | 45.7 |
+|  | EurusPRM-Stage 1 | 77.8 | 44.6 | **26.7** | 35.3 | 41.5 | 45.2 |
+| Weighted Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | **81.2** | **56.6** | 23.3 | **42.4** | 38.2 | 48.3 |
+|  | EurusPRM-Stage 1 | 80.4 | 53.0 | **26.7** | 40.9 | **46.7** | **49.5** |
 
-[More Information Needed]
+**Qwen2.5-7B-Instruct**
 
-## More Information [optional]
+| Method | Reward Model | MATH | AMC | AIME 2024 | OlympiadBench | Minerva Math | Avg |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| Greedy Pass @ 1 | N/A | 73.3 | 47.0 | 13.3 | 39.4 | 35.3 | 41.7 |
+| Majority Voting @ 64 | N/A | 82.0 | 53.0 | 16.7 | 43.0 | 36.4 | 46.2 |
+| Best-of-N @ 64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | **85.2** | **60.2** | **20.0** | **44.7** | 32.7 | 48.6 |
+|  | EurusPRM-Stage 1 | 81.8 | 47.0 | 16.7 | 40.1 | 41.5 | 45.4 |
+|  | EurusPRM-Stage 2 | **86.0** | 59.0 | 16.7 | 41.4 | 41.5 | **48.9** |
+| Weighted Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 83.6 | 55.4 | 13.3 | 43.7 | 36.8 | 46.6 |
+|  | EurusPRM-Stage 1 | 82.6 | 53.0 | 16.7 | 42.7 | **45.2** | 48.0 |
 
-[More Information Needed]
 
-## Model Card Authors [optional]
 
-[More Information Needed]
+## Citation
 
-## Model Card Contact
+```latex
+@misc{cui2024process,
+  title={Process Reinforcement through Implicit Rewards},
+  author={Ganqu Cui and Lifan Yuan and Zefan Wang and Hanbin Wang and Wendi Li and Bingxiang He and Yuchen Fan and Tianyu Yu and Qixin Xu and Weize Chen and Jiarui Yuan and Huayu Chen and Kaiyan Zhang and Xingtai Lv and Shuo Wang and Yuan Yao and Hao Peng and Yu Cheng and Zhiyuan Liu and Maosong Sun and Bowen Zhou and Ning Ding},
+  year={2025}
+}
+```
 
-[More Information Needed]
+```latex
+@article{yuan2024implicitprm,
+  title={Free Process Rewards without Process Labels},
+  author={Lifan Yuan and Wendi Li and Huayu Chen and Ganqu Cui and Ning Ding and Kaiyan Zhang and Bowen Zhou and Zhiyuan Liu and Hao Peng},
+  journal={arXiv preprint arXiv:2412.01981},
+  year={2024}
+}
+```