DeepSeek vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and
  Summarization?
Reasoning-enabled large language models (LLMs) have recently demonstrated
impressive performance in complex logical and mathematical tasks, yet their
effectiveness in evaluating natural language generation remains unexplored.
This study systematically compares reasoning-based LLMs (DeepSeek-R1 and OpenAI
o3) with their non-reasoning counterparts across machine translation (MT) and
text summarization (TS) evaluation tasks. We evaluate eight models across three
architectural categories, including state-of-the-art reasoning models, their
distilled variants (ranging from 8B to 70B parameters), and equivalent
conventional, non-reasoning LLMs. Our experiments on WMT23 and SummEval
benchmarks reveal that the benefits of reasoning capabilities are highly model
and task-dependent: while OpenAI o3-mini models show consistent performance
improvements with increased reasoning intensity, DeepSeek-R1 underperforms
compared to its non-reasoning variant, with exception to certain aspects of TS
evaluation. Correlation analysis demonstrates that increased reasoning token
usage positively correlates with evaluation quality in o3-mini models.
Furthermore, our results show that distillation of reasoning capabilities
maintains reasonable performance in medium-sized models (32B) but degrades
substantially in smaller variants (8B). This work provides the first
comprehensive assessment of reasoning LLMs for NLG evaluation and offers
insights into their practical use.
67fe3a3999bb7d059fc7bd31
Daniil Larionov
claimed_verified
2025-04-16T08:51:57.094Z
60524a44194785248da0894c
https://cdn-avatars.huggingface.co/v1/production/uploads/1636989238554-60524a44194785248da0894c.jpeg
Daniil Larionov
user
Rexhaif
67fe3a3999bb7d059fc7bd32
Sotaro Takeshita
67fe3a3999bb7d059fc7bd33
Ran Zhang
67fe3a3999bb7d059fc7bd34
Yanran Chen
67fe3a3999bb7d059fc7bd35
Christoph Leiter
67fe3a3999bb7d059fc7bd36
Zhipin Wang
67fe3a3999bb7d059fc7bd37
Christian Greisinger
67fe3a3999bb7d059fc7bd38
Steffen Eger
DeepSeek vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and
  Summarization?
Reasoning-enabled large language models (LLMs) have recently demonstrated
impressive performance in complex logical and mathematical tasks, yet their
effectiveness in evaluating natural language generation remains unexplored.
This study systematically compares reasoning-based LLMs (DeepSeek-R1 and OpenAI
o3) with their non-reasoning counterparts across machine translation (MT) and
text summarization (TS) evaluation tasks. We evaluate eight models across three
architectural categories, including state-of-the-art reasoning models, their
distilled variants (ranging from 8B to 70B parameters), and equivalent
conventional, non-reasoning LLMs. Our experiments on WMT23 and SummEval
benchmarks reveal that the benefits of reasoning capabilities are highly model
and task-dependent: while OpenAI o3-mini models show consistent performance
improvements with increased reasoning intensity, DeepSeek-R1 underperforms
compared to its non-reasoning variant, with exception to certain aspects of TS
evaluation. Correlation analysis demonstrates that increased reasoning token
usage positively correlates with evaluation quality in o3-mini models.
Furthermore, our results show that distillation of reasoning capabilities
maintains reasonable performance in medium-sized models (32B) but degrades
substantially in smaller variants (8B). This work provides the first
comprehensive assessment of reasoning LLMs for NLG evaluation and offers
insights into their practical use.
Reasoning-enabled LLMs show mixed performance improvements over non-reasoning models in natural language generation tasks, with performance varying by model and task.
reasoning-enabled large language models DeepSeek-R1 OpenAI o3 machine translation text summarization state-of-the-art reasoning models distilled variants conventional LLMs WMT23 SummEval reasoning intensity reasoning tokens