title: DeepSeek vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and
    Summarization?
summary: Reasoning-enabled large language models (LLMs) have recently demonstrated
  impressive performance in complex logical and mathematical tasks, yet their
  effectiveness in evaluating natural language generation remains unexplored.
  This study systematically compares reasoning-based LLMs (DeepSeek-R1 and OpenAI
  o3) with their non-reasoning counterparts across machine translation (MT) and
  text summarization (TS) evaluation tasks. We evaluate eight models across three
  architectural categories, including state-of-the-art reasoning models, their
  distilled variants (ranging from 8B to 70B parameters), and equivalent
  conventional, non-reasoning LLMs. Our experiments on WMT23 and SummEval
  benchmarks reveal that the benefits of reasoning capabilities are highly model
  and task-dependent: while OpenAI o3-mini models show consistent performance
  improvements with increased reasoning intensity, DeepSeek-R1 underperforms
  compared to its non-reasoning variant, with exception to certain aspects of TS
  evaluation. Correlation analysis demonstrates that increased reasoning token
  usage positively correlates with evaluation quality in o3-mini models.
  Furthermore, our results show that distillation of reasoning capabilities
  maintains reasonable performance in medium-sized models (32B) but degrades
  substantially in smaller variants (8B). This work provides the first
  comprehensive assessment of reasoning LLMs for NLG evaluation and offers
  insights into their practical use.
paper_authors._id: 67fe3a3999bb7d059fc7bd31
paper_authors.name: Daniil Larionov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T08:51:57.094Z
paper_authors._id: 60524a44194785248da0894c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636989238554-60524a44194785248da0894c.jpeg
paper_authors.fullname: Daniil Larionov
paper_authors.type: user
paper_authors.user: Rexhaif
paper_authors._id: 67fe3a3999bb7d059fc7bd32
paper_authors.name: Sotaro Takeshita
paper_authors._id: 67fe3a3999bb7d059fc7bd33
paper_authors.name: Ran Zhang
paper_authors._id: 67fe3a3999bb7d059fc7bd34
paper_authors.name: Yanran Chen
paper_authors._id: 67fe3a3999bb7d059fc7bd35
paper_authors.name: Christoph Leiter
paper_authors._id: 67fe3a3999bb7d059fc7bd36
paper_authors.name: Zhipin Wang
paper_authors._id: 67fe3a3999bb7d059fc7bd37
paper_authors.name: Christian Greisinger
paper_authors._id: 67fe3a3999bb7d059fc7bd38
paper_authors.name: Steffen Eger
paper_title: DeepSeek vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and
    Summarization?
paper_summary: Reasoning-enabled large language models (LLMs) have recently demonstrated
  impressive performance in complex logical and mathematical tasks, yet their
  effectiveness in evaluating natural language generation remains unexplored.
  This study systematically compares reasoning-based LLMs (DeepSeek-R1 and OpenAI
  o3) with their non-reasoning counterparts across machine translation (MT) and
  text summarization (TS) evaluation tasks. We evaluate eight models across three
  architectural categories, including state-of-the-art reasoning models, their
  distilled variants (ranging from 8B to 70B parameters), and equivalent
  conventional, non-reasoning LLMs. Our experiments on WMT23 and SummEval
  benchmarks reveal that the benefits of reasoning capabilities are highly model
  and task-dependent: while OpenAI o3-mini models show consistent performance
  improvements with increased reasoning intensity, DeepSeek-R1 underperforms
  compared to its non-reasoning variant, with exception to certain aspects of TS
  evaluation. Correlation analysis demonstrates that increased reasoning token
  usage positively correlates with evaluation quality in o3-mini models.
  Furthermore, our results show that distillation of reasoning capabilities
  maintains reasonable performance in medium-sized models (32B) but degrades
  substantially in smaller variants (8B). This work provides the first
  comprehensive assessment of reasoning LLMs for NLG evaluation and offers
  insights into their practical use.
paper_ai_summary: Reasoning-enabled LLMs show mixed performance improvements over non-reasoning models in natural language generation tasks, with performance varying by model and task.
paper_ai_keywords: reasoning-enabled large language models DeepSeek-R1 OpenAI o3 machine translation text summarization state-of-the-art reasoning models distilled variants conventional LLMs WMT23 SummEval reasoning intensity reasoning tokens