title: RePOPE: Impact of Annotation Errors on the POPE Benchmark
summary: Since data annotation is costly, benchmark datasets often incorporate labels
  from established image datasets. In this work, we assess the impact of label
  errors in MSCOCO on the frequently used object hallucination benchmark POPE. We
  re-annotate the benchmark images and identify an imbalance in annotation errors
  across different subsets. Evaluating multiple models on the revised labels,
  which we denote as RePOPE, we observe notable shifts in model rankings,
  highlighting the impact of label quality. Code and data are available at
  https://github.com/YanNeu/RePOPE .
paper_authors._id: 6809f6b03e1d48a9bb7f5719
paper_authors.name: Yannic Neuhaus
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-24T09:09:36.629Z
paper_authors._id: 650971dbce83a0c12a851000
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ec7H2cZpaZz07W02FT5U9.png
paper_authors.fullname: Yannic Neuhaus
paper_authors.type: user
paper_authors.user: YanNeu
paper_authors._id: 6809f6b03e1d48a9bb7f571a
paper_authors.name: Matthias Hein
paper_title: RePOPE: Impact of Annotation Errors on the POPE Benchmark
paper_summary: Since data annotation is costly, benchmark datasets often incorporate labels
  from established image datasets. In this work, we assess the impact of label
  errors in MSCOCO on the frequently used object hallucination benchmark POPE. We
  re-annotate the benchmark images and identify an imbalance in annotation errors
  across different subsets. Evaluating multiple models on the revised labels,
  which we denote as RePOPE, we observe notable shifts in model rankings,
  highlighting the impact of label quality. Code and data are available at
  https://github.com/YanNeu/RePOPE .
paper_ai_summary: Revising labels in the POPE benchmark dataset reveals significant shifts in model performance, emphasizing the importance of label quality.
paper_ai_keywords: label errors MSCOCO object hallucination benchmark POPE RePOPE