Profiling News Media for Factuality and Bias Using LLMs and the
  Fact-Checking Methodology of Human Experts
In an age characterized by the proliferation of mis- and disinformation
online, it is critical to empower readers to understand the content they are
reading. Important efforts in this direction rely on manual or automatic
fact-checking, which can be challenging for emerging claims with limited
information. Such scenarios can be handled by assessing the reliability and the
political bias of the source of the claim, i.e., characterizing entire news
outlets rather than individual claims or articles. This is an important but
understudied research direction. While prior work has looked into linguistic
and social contexts, we do not analyze individual articles or information in
social media. Instead, we propose a novel methodology that emulates the
criteria that professional fact-checkers use to assess the factuality and
political bias of an entire outlet. Specifically, we design a variety of
prompts based on these criteria and elicit responses from large language models
(LLMs), which we aggregate to make predictions. In addition to demonstrating
sizable improvements over strong baselines via extensive experiments with
multiple LLMs, we provide an in-depth error analysis of the effect of media
popularity and region on model performance. Further, we conduct an ablation
study to highlight the key components of our dataset that contribute to these
improvements. To facilitate future research, we released our dataset and code
at https://github.com/mbzuai-nlp/llm-media-profiling.
6851336a8a68fee7f6ba4c0b
Zain Muhammad Mujahid
claimed_verified
2025-06-17T13:06:26.481Z
637e8b1b66ee00bcb2468ed0
https://cdn-avatars.huggingface.co/v1/production/uploads/1669240174964-637e8b1b66ee00bcb2468ed0.jpeg
Zain
user
zainmujahid
6851336a8a68fee7f6ba4c0c
Dilshod Azizov
6851336a8a68fee7f6ba4c0d
Maha Tufail Agro
6851336a8a68fee7f6ba4c0e
Preslav Nakov
claimed_verified
2025-07-09T15:10:31.573Z
647f7eb25e1bc4753746bf9f
/avatars/cc9c6210fdc822d8a106937e747dff41.svg
Preslav Nakov
user
preslavnakov
Profiling News Media for Factuality and Bias Using LLMs and the
  Fact-Checking Methodology of Human Experts
In an age characterized by the proliferation of mis- and disinformation
online, it is critical to empower readers to understand the content they are
reading. Important efforts in this direction rely on manual or automatic
fact-checking, which can be challenging for emerging claims with limited
information. Such scenarios can be handled by assessing the reliability and the
political bias of the source of the claim, i.e., characterizing entire news
outlets rather than individual claims or articles. This is an important but
understudied research direction. While prior work has looked into linguistic
and social contexts, we do not analyze individual articles or information in
social media. Instead, we propose a novel methodology that emulates the
criteria that professional fact-checkers use to assess the factuality and
political bias of an entire outlet. Specifically, we design a variety of
prompts based on these criteria and elicit responses from large language models
(LLMs), which we aggregate to make predictions. In addition to demonstrating
sizable improvements over strong baselines via extensive experiments with
multiple LLMs, we provide an in-depth error analysis of the effect of media
popularity and region on model performance. Further, we conduct an ablation
study to highlight the key components of our dataset that contribute to these
improvements. To facilitate future research, we released our dataset and code
at https://github.com/mbzuai-nlp/llm-media-profiling.
A novel methodology using large language models with curated prompts improves predictions of media outlet factuality and political bias, validated through experiments and error analysis.
large language models LLMs