title: SelfEval: Leveraging the discriminative nature of generative models for
    evaluation
summary: In this work, we show that text-to-image generative models can be 'inverted'
  to assess their own text-image understanding capabilities in a completely
  automated manner.
    Our method, called SelfEval, uses the generative model to compute the
  likelihood of real images given text prompts, making the generative model
  directly applicable to discriminative tasks.
    Using SelfEval, we repurpose standard datasets created for evaluating
  multimodal text-image discriminative models to evaluate generative models in a
  fine-grained manner: assessing their performance on attribute binding, color
  recognition, counting, shape recognition, spatial understanding.
    To the best of our knowledge SelfEval is the first automated metric to show a
  high degree of agreement for measuring text-faithfulness with the gold-standard
  human evaluations across multiple models and benchmarks.
    Moreover, SelfEval enables us to evaluate generative models on challenging
  tasks such as Winoground image-score where they demonstrate competitive
  performance to discriminative models.
    We also show severe drawbacks of standard automated metrics such as
  CLIP-score to measure text faithfulness on benchmarks such as DrawBench, and
  how SelfEval sidesteps these issues.
    We hope SelfEval enables easy and reliable automated evaluation for diffusion
  models.
paper_authors._id: 655ac665deee83130a470af9
paper_authors.name: Sai Saketh Rambhatla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T09:29:38.103Z
paper_authors._id: 6435a89b876d46610bf30b21
paper_authors.avatarUrl: /avatars/88dba6589b07ad310bad94108774d893.svg
paper_authors.fullname: Saketh Rambhatla
paper_authors.type: user
paper_authors.user: rssaketh
paper_authors._id: 655ac665deee83130a470afa
paper_authors.name: Ishan Misra
paper_title: SelfEval: Leveraging the discriminative nature of generative models for
    evaluation
paper_summary: In this work, we show that text-to-image generative models can be 'inverted'
  to assess their own text-image understanding capabilities in a completely
  automated manner.
    Our method, called SelfEval, uses the generative model to compute the
  likelihood of real images given text prompts, making the generative model
  directly applicable to discriminative tasks.
    Using SelfEval, we repurpose standard datasets created for evaluating
  multimodal text-image discriminative models to evaluate generative models in a
  fine-grained manner: assessing their performance on attribute binding, color
  recognition, counting, shape recognition, spatial understanding.
    To the best of our knowledge SelfEval is the first automated metric to show a
  high degree of agreement for measuring text-faithfulness with the gold-standard
  human evaluations across multiple models and benchmarks.
    Moreover, SelfEval enables us to evaluate generative models on challenging
  tasks such as Winoground image-score where they demonstrate competitive
  performance to discriminative models.
    We also show severe drawbacks of standard automated metrics such as
  CLIP-score to measure text faithfulness on benchmarks such as DrawBench, and
  how SelfEval sidesteps these issues.
    We hope SelfEval enables easy and reliable automated evaluation for diffusion
  models.
paper_ai_keywords: text-to-image generative models SelfEval likelihood discriminative tasks attribute binding color recognition counting shape recognition spatial understanding text-faithfulness Winoground image-score CLIP-score DrawBench diffusion models