Update model

README.md

@@ -1,193 +1,199 @@
 ---
-license: apache-2.0
+library_name: transformers
+tags: []
 ---
-# EurusPRM-Stage2
 
-## Links
+# Model Card for Model ID
 
-- 📜 [Blog](https://curvy-check-498.notion.site/Process-Reinforcement-through-Implicit-Rewards-15f4fcb9c42180f1b498cc9b2eaf896f)
-- 🤗 [PRIME Collection](https://huggingface.co/PRIME-RL)
-- 🤗 [Training Data](https://huggingface.co/datasets/PRIME-RL/EurusPRM-Stage1-Data)
+<!-- Provide a quick summary of what the model is/does. -->
 
-## Introduction
 
-EurusPRM-Stage1 is trained using **[Implicit PRM](https://arxiv.org/abs/2412.01981)**, which obtains free process rewards at no additional cost but just needs to simply train an ORM on the cheaper response-level labels. During inference, implicit process rewards are obtained by forward passing and calculating the log-likelihood ratio on each step. It serves a great fundation for further training of **[EurusPRM-Stage2](https://huggingface.co/PRIME-RL/EurusPRM-Stage2)**.
 
-<img src="./figs/implicit.png" alt="prm" style="zoom: 33%;" />
+## Model Details
 
-The key ingredient of Implicit PRM is the reward representation, as demonstrated below:
+### Model Description
 
-<aside>
-✨
+<!-- Provide a longer summary of what this model is. -->
 
-***Proposition***: Consider an ORM where the reward is parameterized by the log-likelihood ratio of two causal LMs, i.e.
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
 
-$$
-r_\phi(\mathbf{y}) := \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})}.
-$$
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
-Define
+### Model Sources [optional]
 
-$$
-q_\phi^t(\mathbf{y}_{<t}, y_t) := \sum_{i=1}^{t} \beta \log \frac{\pi_\phi(y_{i}|\mathbf{y}_{<i})}{\pi_\text{ref}(y_{i}|\mathbf{y}_{<i})}.
-$$
+<!-- Provide the basic links for the model. -->
 
-is the exponential average of \\(r_\theta\\) at step \\(t\\).
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
 
-$$
-q_\phi^t(\mathbf{y}_{<t}, y_t) = \beta \log \mathbb{E}{\pi_\text{ref}(\mathbf{y}|\mathbf{y}_{\leq t})} \left[ e^{\frac{1}{\beta} r_\phi(\mathbf{y})} \right]
-$$
+## Uses
 
-Hence, \\(q_\theta^t\\)represents an exact expectation of outcome reward \\(r_\theta\\) at step \\(t\\), i.e., the Q value.
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-The proposition indicates that when modeling
+### Direct Use
 
-$$
-r_\phi(\mathbf{y}) := \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})}
-$$
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-to train an ORM with the standard pipeline, where \\(\beta\\) is a hyperparameter, \\(\phi\\) can implicitly learn a Q function. Hence, process reward \\(r_\phi^t\\) can be obtained by:
+[More Information Needed]
 
-$$
-r_\phi^t := q_\phi^t - q_\phi^{t-1} = \beta \log \frac{\pi_\phi(y_{t}|\mathbf{y}_{<t})}{\pi_\text{ref}(y_{t}|\mathbf{y}_{<t})}.
-$$
+### Downstream Use [optional]
 
-Therefore, we can indeed obtain PRMs simply by collecting response-level data and training an ORM, without any burden of annotating step labels.
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-The proposition is **agnostic to specific choices of the training objective of ORMs**. It can be instantiated with different objectives as vanilla ORM training, with the only difference being substituting the \\(r_\phi \left( \mathbf{y} \right)\\) with \\(\beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})}\\).
+[More Information Needed]
 
-For example, DPO already meets our assumption and serves as a strong variant, while in this work, we instantiate our implicit PRM with cross entropy (CE) loss due to memory efficiency:
+### Out-of-Scope Use
 
-$$
-\small \mathcal{L}_{CE} = l \cdot \log \sigma \left( \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})} \right) + (1 - l) \cdot \log \left[ 1 - \sigma \left( \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})} \right) \right]
-$$
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-We applied \\(L_{CE}\\) to train implicit PRM. We used a learning rate of 5e-7 and a batch-size of 64 for training. 
+[More Information Needed]
 
-## Usage
+## Bias, Risks, and Limitations
 
-We show an example leveraging **EurusPRM-Stage1** below:
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-```python
-import torch
-from transformers import AutoTokenizer,AutoModelForCausalLM
-coef=0.001
-d = {'query':'Convert the point $(0,3)$ in rectangular coordinates to polar coordinates.  Enter your answer in the form $(r,\\theta),$ where $r > 0$ and $0 \\le \\theta < 2 \\pi.$',
-     'answer':[
-"Step 1: To convert the point (0,3) from rectangular coordinates to polar coordinates, we need to find the radius (r) and the angle theta (\u03b8).",
-            "Step 1: Find the radius (r). The radius is the distance from the origin (0,0) to the point (0,3). Since the x-coordinate is 0, the distance is simply the absolute value of the y-coordinate. So, r = |3| = 3.",
-            "Step 2: Find the angle theta (\u03b8). The angle theta is measured counterclockwise from the positive x-axis. Since the point (0,3) lies on the positive y-axis, the angle theta is 90 degrees or \u03c0\/2 radians.",
-            "Step 3: Write the polar coordinates. The polar coordinates are (r, \u03b8), where r > 0 and 0 \u2264 \u03b8 < 2\u03c0. In this case, r = 3 and \u03b8 = \u03c0\/2.\n\nTherefore, the polar coordinates of the point (0,3) are (3, \u03c0\/2).\n\n\n\\boxed{(3,\\frac{\\pi}{2})}"
-     ]
-     }
-model = AutoModelForCausalLM.from_pretrained('PRIME-RL/EurusPRM-Stage1')
-tokenizer = AutoTokenizer.from_pretrained('PRIME-RL/EurusPRM-Stage1')
-ref_model = AutoModelForCausalLM.from_pretrained('Qwen/Qwen2.5-Math-7B-Instruct')
-input_ids = tokenizer.apply_chat_template([
-    {"role": "user", "content": d["query"]},
-    {"role": "assistant", "content": "\n\n".join(d["answer"])},
-], tokenize=True, add_generation_prompt=False,return_tensors='pt')
-attention_mask = input_ids!=tokenizer.pad_token_id
-step_last_tokens = []
-for step_num in range(0, len(d["answer"])+1):
-    conv = tokenizer.apply_chat_template([
-        {"role":"user", "content":d["query"]},
-        {"role":"assistant", "content":"\n\n".join(d["answer"][:step_num])},
-    ], tokenize=False, add_generation_prompt=False)
-    conv = conv.strip()
-    if step_num!=0 and step_num!=len(d['answer']):
-        conv+='\n\n'
-    currect_ids = tokenizer.encode(conv,add_special_tokens=False)
-    step_last_tokens.append(len(currect_ids) - 2)
+[More Information Needed]
 
+### Recommendations
 
-inputs = {'input_ids':input_ids,'attention_mask':attention_mask,'labels':input_ids}
-label_mask = torch.tensor([[0]*step_last_tokens[0]+[1]*(input_ids.shape[-1]-step_last_tokens[0])])
-step_last_tokens = torch.tensor([step_last_tokens])
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-def get_logps(model,inputs):
-    logits = model(input_ids=inputs['input_ids'], attention_mask=inputs['attention_mask']).logits
-    labels = inputs['labels'][:, 1:].clone().long()
-    logits = logits[:, :-1, :]
-    labels[labels == -100] = 0
-    per_token_logps = torch.gather(logits.log_softmax(-1), dim=2, index=labels.unsqueeze(2)).squeeze(2)
-    return per_token_logps
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-with torch.no_grad():
-    per_token_logps = get_logps(model, inputs)
-    ref_per_token_logps = get_logps(ref_model,inputs)
+## How to Get Started with the Model
 
-raw_reward = per_token_logps - ref_per_token_logps
-beta_reward = coef * raw_reward * label_mask[:,1:]
-beta_reward = beta_reward.cumsum(-1)
-beta_reward = beta_reward.gather(dim=-1, index=step_last_tokens[:,1:])
-
-print(beta_reward)
-```
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
 
 ## Evaluation
 
-### Evaluation Base Model
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
 
-We adopt **Eurus-2-7B-SFT**, **Qwen2.5-7B-Instruct** and **Llama-3.1-70B-Instruct** as generation models to evaluate the performance of our implicit PRM. For all models, we set the sampling temperature as 0.5, *p* of the top-*p* sampling as 1.
+## Citation [optional]
 
-### Best-of-N Sampling
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
-We use Best-of-64 as our evaluation metric. The weighting methods are different for several PRMs below.
+**BibTeX:**
 
-- For [Skywork-o1-Open-PRM-Qwen-2.5-7B](https://huggingface.co/Skywork/Skywork-o1-Open-PRM-Qwen-2.5-7B), we use simple average reward across all steps.
-- For EurusPRM-Stage 1, we use the minimum reward across all steps.
+[More Information Needed]
 
-**Eurus-2-7B-SFT**
+**APA:**
 
-| Method | Reward Model | MATH | AMC | AIME_2024 | OlympiadBench | Minerva Math | Avg |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| Greedy Pass @ 1 | N/A | 65.1 | 30.1 | 3.3 | 29.8 | 32.7 | 32.2 |
-| Majority Voting @ 64 | N/A | 65.6 | 53.0 | 13.3 | 39.1 | 22.4 | 38.7 |
-| Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 47.2 | 45.8 | 10.0 | 32.3 | 16.2 | 30.3 |
-|  | EurusPRM-Stage 1 | 44.6 | 41.0 | 6.7 | 32.9 | 17.3 | 28.5 |
-| Weighted Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 64.6 | **55.4** | 13.3 | **41.3** | 23.2 | 39.6 |
-|  | EurusPRM-Stage 1 | **66.0** | 54.2 | **13.3** | 39.6 | **29.0** | **40.4** |
+[More Information Needed]
 
-**Llama-3.1-70B-Instruct**
+## Glossary [optional]
 
-| Method | Reward Model | MATH | AMC | AIME 2024 | OlympiadBench | Minerva Math | Avg |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| Greedy Pass @ 1 | N/A | 64.6 | 30.1 | 16.7 | 31.9 | 35.3 | 35.7 |
-| Majority Voting @ 64 | N/A | 80.2 | 53.0 | 26.7 | 40.4 | 38.6 | 47.8 |
-| Best-of-N @ 64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 77.8 | 56.6 | 23.3 | 39.0 | 31.6 | 45.7 |
-|  | EurusPRM-Stage 1 | 77.8 | 44.6 | **26.7** | 35.3 | 41.5 | 45.2 |
-| Weighted Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | **81.2** | **56.6** | 23.3 | **42.4** | 38.2 | 48.3 |
-|  | EurusPRM-Stage 1 | 80.4 | 53.0 | **26.7** | 40.9 | **46.7** | **49.5** |
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 
-**Qwen2.5-7B-Instruct**
+[More Information Needed]
 
-| Method | Reward Model | MATH | AMC | AIME 2024 | OlympiadBench | Minerva Math | Avg |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| Greedy Pass @ 1 | N/A | 73.3 | 47.0 | 13.3 | 39.4 | 35.3 | 41.7 |
-| Majority Voting @ 64 | N/A | 82.0 | 53.0 | 16.7 | 43.0 | 36.4 | 46.2 |
-| Best-of-N @ 64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | **85.2** | **60.2** | **20.0** | **44.7** | 32.7 | **48.6** |
-|  | EurusPRM-Stage 1 | 81.8 | 47.0 | 16.7 | 40.1 | 41.5 | 45.4 |
-| Weighted Best-of-64 | Skywork-o1-Open-PRM-Qwen-2.5-7B | 83.6 | 55.4 | 13.3 | 43.7 | 36.8 | 46.6 |
-|  | EurusPRM-Stage 1 | 82.6 | 53.0 | 16.7 | 42.7 | **45.2** | 48.0 |
+## More Information [optional]
 
+[More Information Needed]
 
+## Model Card Authors [optional]
 
-## Citation
+[More Information Needed]
 
-```latex
-@misc{cui2024process,
-  title={Process Reinforcement through Implicit Rewards},
-  author={Ganqu Cui and Lifan Yuan and Zefan Wang and Hanbin Wang and Wendi Li and Bingxiang He and Yuchen Fan and Tianyu Yu and Qixin Xu and Weize Chen and Jiarui Yuan and Huayu Chen and Kaiyan Zhang and Xingtai Lv and Shuo Wang and Yuan Yao and Hao Peng and Yu Cheng and Zhiyuan Liu and Maosong Sun and Bowen Zhou and Ning Ding},
-  year={2025}
-}
-```
+## Model Card Contact
 
-```latex
-@article{yuan2024implicitprm,
-  title={Free Process Rewards without Process Labels},
-  author={Lifan Yuan and Wendi Li and Huayu Chen and Ganqu Cui and Ning Ding and Kaiyan Zhang and Bowen Zhou and Zhiyuan Liu and Hao Peng},
-  journal={arXiv preprint arXiv:2412.01981},
-  year={2024}
-}
-```
+[More Information Needed]