title: Unraveling the Capabilities of Language Models in News Summarization
summary: Given the recent introduction of multiple language models and the ongoing
  demand for improved Natural Language Processing tasks, particularly
  summarization, this work provides a comprehensive benchmarking of 20 recent
  language models, focusing on smaller ones for the news summarization task. In
  this work, we systematically test the capabilities and effectiveness of these
  models in summarizing news article texts which are written in different styles
  and presented in three distinct datasets. Specifically, we focus in this study
  on zero-shot and few-shot learning settings and we apply a robust evaluation
  methodology that combines different evaluation concepts including automatic
  metrics, human evaluation, and LLM-as-a-judge. Interestingly, including
  demonstration examples in the few-shot learning setting did not enhance models'
  performance and, in some cases, even led to worse quality of the generated
  summaries. This issue arises mainly due to the poor quality of the gold
  summaries that have been used as reference summaries, which negatively impacts
  the models' performance. Furthermore, our study's results highlight the
  exceptional performance of GPT-3.5-Turbo and GPT-4, which generally dominate
  due to their advanced capabilities. However, among the public models evaluated,
  certain models such as Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B
  and Zephyr-7B-Beta demonstrated promising results. These models showed
  significant potential, positioning them as competitive alternatives to large
  models for the task of news summarization.
paper_authors._id: 679e04b792d873dfa23d0ba6
paper_authors.name: Abdurrahman Odabaşı
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T08:14:51.873Z
paper_authors._id: 647d79a736e109abce419102
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647d79a736e109abce419102/S8Hby6eO4WdPQrct0Ix3c.png
paper_authors.fullname: Abdurrahman Odabaşı
paper_authors.type: user
paper_authors.user: odabashi
paper_authors._id: 679e04b792d873dfa23d0ba7
paper_authors.name: Göksel Biricik
paper_title: Unraveling the Capabilities of Language Models in News Summarization
paper_summary: Given the recent introduction of multiple language models and the ongoing
  demand for improved Natural Language Processing tasks, particularly
  summarization, this work provides a comprehensive benchmarking of 20 recent
  language models, focusing on smaller ones for the news summarization task. In
  this work, we systematically test the capabilities and effectiveness of these
  models in summarizing news article texts which are written in different styles
  and presented in three distinct datasets. Specifically, we focus in this study
  on zero-shot and few-shot learning settings and we apply a robust evaluation
  methodology that combines different evaluation concepts including automatic
  metrics, human evaluation, and LLM-as-a-judge. Interestingly, including
  demonstration examples in the few-shot learning setting did not enhance models'
  performance and, in some cases, even led to worse quality of the generated
  summaries. This issue arises mainly due to the poor quality of the gold
  summaries that have been used as reference summaries, which negatively impacts
  the models' performance. Furthermore, our study's results highlight the
  exceptional performance of GPT-3.5-Turbo and GPT-4, which generally dominate
  due to their advanced capabilities. However, among the public models evaluated,
  certain models such as Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B
  and Zephyr-7B-Beta demonstrated promising results. These models showed
  significant potential, positioning them as competitive alternatives to large
  models for the task of news summarization.
paper_ai_summary: A benchmark of smaller language models for news summarization in zero-shot and few-shot settings highlights GPT-3.5-Turbo and GPT-4 as top performers, while also identifying Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B, and Zephyr-7B-Beta as promising alternatives.
paper_ai_keywords: zero-shot few-shot learning LLM-as-a-judge GPT-3.5-Turbo GPT-4 Qwen1.5-7B SOLAR-10.7B-Instruct-v1.0 Meta-Llama-3-8B Zephyr-7B-Beta