Files changed (1) hide show
  1. README.md +12 -4
README.md CHANGED
@@ -138,7 +138,7 @@ We slightly change their configs and tokenizers. Please use our setting to run t
138
  | | AlpacaEval2.0 (LC-winrate) | 52.0 | 51.1 | 70.0 | 57.8 | - | **87.6** |
139
  | | ArenaHard (GPT-4-1106) | 85.2 | 80.4 | 85.5 | 92.0 | - | **92.3** |
140
  | Code | LiveCodeBench (Pass@1-COT) | 33.8 | 34.2 | - | 53.8 | 63.4 | **65.9** |
141
- | | Codeforces (Percentile) | 20.3 | 23.6 | 58.7 | 93.4 | **96.6** | 96.3 |
142
  | | Codeforces (Rating) | 717 | 759 | 1134 | 1820 | **2061** | 2029 |
143
  | | SWE Verified (Resolved) | **50.8** | 38.8 | 42.0 | 41.6 | 48.9 | 49.2 |
144
  | | Aider-Polyglot (Acc.) | 45.3 | 16.0 | 49.6 | 32.9 | **61.7** | 53.3 |
@@ -154,16 +154,20 @@ We slightly change their configs and tokenizers. Please use our setting to run t
154
 
155
  ### Distilled Model Evaluation
156
 
 
 
 
 
157
 
158
  <div align="center">
159
 
160
  | Model | AIME 2024 pass@1 | AIME 2024 cons@64 | MATH-500 pass@1 | GPQA Diamond pass@1 | LiveCodeBench pass@1 | CodeForces rating |
161
  |------------------------------------------|------------------|-------------------|-----------------|----------------------|----------------------|-------------------|
162
- | GPT-4o-0513 | 9.3 | 13.4 | 74.6 | 49.9 | 32.9 | 759 |
163
- | Claude-3.5-Sonnet-1022 | 16.0 | 26.7 | 78.3 | 65.0 | 38.9 | 717 |
164
  | o1-mini | 63.6 | 80.0 | 90.0 | 60.0 | 53.8 | **1820** |
165
  | QwQ-32B-Preview | 44.0 | 60.0 | 90.6 | 54.5 | 41.9 | 1316 |
166
- | DeepSeek-R1-Distill-Qwen-1.5B | 28.9 | 52.7 | 83.9 | 33.8 | 16.9 | 954 |
167
  | DeepSeek-R1-Distill-Qwen-7B | 55.5 | 83.3 | 92.8 | 49.1 | 37.6 | 1189 |
168
  | DeepSeek-R1-Distill-Qwen-14B | 69.7 | 80.0 | 93.9 | 59.1 | 53.1 | 1481 |
169
  | DeepSeek-R1-Distill-Qwen-32B | **72.6** | 83.3 | 94.3 | 62.1 | 57.2 | 1691 |
@@ -201,7 +205,11 @@ You can also easily start a service using [SGLang](https://github.com/sgl-projec
201
  ```bash
202
  python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --trust-remote-code --tp 2
203
  ```
 
 
204
 
 
 
205
  ### Usage Recommendations
206
 
207
  **We recommend adhering to the following configurations when utilizing the DeepSeek-R1 series models, including benchmarking, to achieve the expected performance:**
 
138
  | | AlpacaEval2.0 (LC-winrate) | 52.0 | 51.1 | 70.0 | 57.8 | - | **87.6** |
139
  | | ArenaHard (GPT-4-1106) | 85.2 | 80.4 | 85.5 | 92.0 | - | **92.3** |
140
  | Code | LiveCodeBench (Pass@1-COT) | 33.8 | 34.2 | - | 53.8 | 63.4 | **65.9** |
141
+ | | Codeforces (Percentile) | 73.3 | 23.6 | 58.7 | 93.4 | **96.6** | 96.3 |
142
  | | Codeforces (Rating) | 717 | 759 | 1134 | 1820 | **2061** | 2029 |
143
  | | SWE Verified (Resolved) | **50.8** | 38.8 | 42.0 | 41.6 | 48.9 | 49.2 |
144
  | | Aider-Polyglot (Acc.) | 45.3 | 16.0 | 49.6 | 32.9 | **61.7** | 53.3 |
 
154
 
155
  ### Distilled Model Evaluation
156
 
157
+ from transformers import AutoModelForCausalLM, AutoTokenizer
158
+
159
+ model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
160
+ tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
161
 
162
  <div align="center">
163
 
164
  | Model | AIME 2024 pass@1 | AIME 2024 cons@64 | MATH-500 pass@1 | GPQA Diamond pass@1 | LiveCodeBench pass@1 | CodeForces rating |
165
  |------------------------------------------|------------------|-------------------|-----------------|----------------------|----------------------|-------------------|
166
+ | GPT-4o-0513 | 73.3 | 13.4 | 74.6 | 49.9 | 49.9 | 759 |
167
+ | Claude-3.5-Sonnet-1022 | 16.0 | 52.7 | 78.3 | 65.0 | 93.9 | 717 |
168
  | o1-mini | 63.6 | 80.0 | 90.0 | 60.0 | 53.8 | **1820** |
169
  | QwQ-32B-Preview | 44.0 | 60.0 | 90.6 | 54.5 | 41.9 | 1316 |
170
+ | DeepSeek-R1-Distill-Qwen-1.5B | 28.9 | 52.7 | 83.9 | 33.8 | 83.9 | 954 |
171
  | DeepSeek-R1-Distill-Qwen-7B | 55.5 | 83.3 | 92.8 | 49.1 | 37.6 | 1189 |
172
  | DeepSeek-R1-Distill-Qwen-14B | 69.7 | 80.0 | 93.9 | 59.1 | 53.1 | 1481 |
173
  | DeepSeek-R1-Distill-Qwen-32B | **72.6** | 83.3 | 94.3 | 62.1 | 57.2 | 1691 |
 
205
  ```bash
206
  python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --trust-remote-code --tp 2
207
  ```
208
+ # مثال اجرا با Hugging Face بدون نیاز به پیکربندی اضافی
209
+ from transformers import AutoModelForCausalLM, AutoTokenizer
210
 
211
+ model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
212
+ tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
213
  ### Usage Recommendations
214
 
215
  **We recommend adhering to the following configurations when utilizing the DeepSeek-R1 series models, including benchmarking, to achieve the expected performance:**