title: LANCE: Stress-testing Visual Models by Generating Language-guided
    Counterfactual Images
summary: We propose an automated algorithm to stress-test a trained visual model by
  generating language-guided counterfactual test images (LANCE). Our method
  leverages recent progress in large language modeling and text-based image
  editing to augment an IID test set with a suite of diverse, realistic, and
  challenging test images without altering model weights. We benchmark the
  performance of a diverse set of pretrained models on our generated data and
  observe significant and consistent performance drops. We further analyze model
  sensitivity across different types of edits, and demonstrate its applicability
  at surfacing previously unknown class-level model biases in ImageNet.
paper_authors._id: 6476c6606ab485374ba99c97
paper_authors.name: Viraj Prabhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T16:54:26.058Z
paper_authors._id: 63fe4f92c0ec83fda436b6bf
paper_authors.avatarUrl: /avatars/1021060e042ce3257444577724aa9d34.svg
paper_authors.fullname: Viraj Prabhu
paper_authors.type: user
paper_authors.user: virajp
paper_authors._id: 6476c6606ab485374ba99c98
paper_authors.name: Sriram Yenamandra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T05:54:27.298Z
paper_authors._id: 640d3ece4b5450a208b275fd
paper_authors.avatarUrl: /avatars/669e161448c0895b856065646d68c24c.svg
paper_authors.fullname: sriram
paper_authors.type: user
paper_authors.user: yvsriram
paper_authors._id: 6476c6606ab485374ba99c99
paper_authors.name: Prithvijit Chattopadhyay
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T16:53:55.893Z
paper_authors._id: 628d451386d23ad1560882c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628d451386d23ad1560882c4/UMxez0DEvX5qdP5ddqi-8.png
paper_authors.fullname: Prithvijit Chattopadhyay
paper_authors.type: user
paper_authors.user: prithv1
paper_authors._id: 6476c6606ab485374ba99c9a
paper_authors.name: Judy Hoffman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T16:53:38.940Z
paper_authors._id: 6477790c4ae93470ffc297bb
paper_authors.avatarUrl: /avatars/f8d5e08719297a335731a117416d6336.svg
paper_authors.fullname: Judy Hoffman
paper_authors.type: user
paper_authors.user: judyhoffman
paper_title: LANCE: Stress-testing Visual Models by Generating Language-guided
    Counterfactual Images
paper_summary: We propose an automated algorithm to stress-test a trained visual model by
  generating language-guided counterfactual test images (LANCE). Our method
  leverages recent progress in large language modeling and text-based image
  editing to augment an IID test set with a suite of diverse, realistic, and
  challenging test images without altering model weights. We benchmark the
  performance of a diverse set of pretrained models on our generated data and
  observe significant and consistent performance drops. We further analyze model
  sensitivity across different types of edits, and demonstrate its applicability
  at surfacing previously unknown class-level model biases in ImageNet.
paper_ai_summary: A new algorithm generates language-guided counterfactual images to stress-test visual models, revealing performance drops and class-level biases.
paper_ai_keywords: language-guided counterfactual test images large language modeling text-based image editing pretrained models class-level model biases