title: TrueTeacher: Learning Factual Consistency Evaluation with Large Language
    Models
summary: Factual consistency evaluation is often conducted using Natural Language
  Inference (NLI) models, yet these models exhibit limited success in evaluating
  summaries. Previous work improved such models with synthetic training data.
  However, the data is typically based on perturbed human-written summaries,
  which often differ in their characteristics from real model-generated summaries
  and have limited coverage of possible factual errors. Alternatively, large
  language models (LLMs) have recently shown promising results in directly
  evaluating generative tasks, but are too computationally expensive for
  practical use. Motivated by these limitations, we introduce TrueTeacher, a
  method for generating synthetic data by annotating diverse model-generated
  summaries using a LLM. Unlike prior work, TrueTeacher does not rely on
  human-written summaries, and is multilingual by nature. Experiments on the TRUE
  benchmark show that a student model trained using our data, substantially
  outperforms both the state-of-the-art model with similar capacity, and the LLM
  teacher. In a systematic study, we compare TrueTeacher to existing synthetic
  data generation methods and demonstrate its superiority and robustness to
  domain-shift. Using the the mFACE dataset, we also show that our method
  generalizes to multilingual scenarios. Finally, we release a large-scale
  synthetic dataset with 1.4M examples generated using TrueTeacher.
paper_authors._id: 6466fc375993aa7666cc0a5b
paper_authors.name: Zorik Gekhman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T14:33:40.975Z
paper_authors._id: 61499a8f8517f77f09a3a990
paper_authors.avatarUrl: /avatars/fa15d117e8ce8c50669455d3e483408b.svg
paper_authors.fullname: Zorik
paper_authors.type: user
paper_authors.user: zorik
paper_authors._id: 6466fc375993aa7666cc0a5c
paper_authors.name: Jonathan Herzig
paper_authors._id: 6466fc375993aa7666cc0a5d
paper_authors.name: Roee Aharoni
paper_authors._id: 6466fc375993aa7666cc0a5e
paper_authors.name: Chen Elkind
paper_authors._id: 6466fc375993aa7666cc0a5f
paper_authors.name: Idan Szpektor
paper_title: TrueTeacher: Learning Factual Consistency Evaluation with Large Language
    Models
paper_summary: Factual consistency evaluation is often conducted using Natural Language
  Inference (NLI) models, yet these models exhibit limited success in evaluating
  summaries. Previous work improved such models with synthetic training data.
  However, the data is typically based on perturbed human-written summaries,
  which often differ in their characteristics from real model-generated summaries
  and have limited coverage of possible factual errors. Alternatively, large
  language models (LLMs) have recently shown promising results in directly
  evaluating generative tasks, but are too computationally expensive for
  practical use. Motivated by these limitations, we introduce TrueTeacher, a
  method for generating synthetic data by annotating diverse model-generated
  summaries using a LLM. Unlike prior work, TrueTeacher does not rely on
  human-written summaries, and is multilingual by nature. Experiments on the TRUE
  benchmark show that a student model trained using our data, substantially
  outperforms both the state-of-the-art model with similar capacity, and the LLM
  teacher. In a systematic study, we compare TrueTeacher to existing synthetic
  data generation methods and demonstrate its superiority and robustness to
  domain-shift. Using the the mFACE dataset, we also show that our method
  generalizes to multilingual scenarios. Finally, we release a large-scale
  synthetic dataset with 1.4M examples generated using TrueTeacher.
paper_ai_summary: TrueTeacher, a method for generating synthetic data using a large language model, improves factual consistency evaluation of summaries compared to existing models and synthetic data techniques.
paper_ai_keywords: Natural Language Inference (NLI) large language models (LLMs) synthetic data model-generated summaries factual consistency evaluation TrueTeacher student model TRUE benchmark mFACE dataset multilingual scenarios