title: Modular Visual Question Answering via Code Generation
summary: We present a framework that formulates visual question answering as modular
  code generation. In contrast to prior work on modular approaches to VQA, our
  approach requires no additional training and relies on pre-trained language
  models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA
  examples used for in-context learning. The generated Python programs invoke and
  compose the outputs of the visual models using arithmetic and conditional
  logic. Our approach improves accuracy on the COVR dataset by at least 3% and on
  the GQA dataset by roughly 2% compared to the few-shot baseline that does not
  employ code generation.
paper_authors._id: 64829bf6e4bbb1c2dd35da74
paper_authors.name: Sanjay Subramanian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:34:58.858Z
paper_authors._id: 62cd947a3a2cecfdabeac91f
paper_authors.avatarUrl: /avatars/84393f357c0370edbb90a4188df795d6.svg
paper_authors.fullname: Sanjay Subramanian
paper_authors.type: user
paper_authors.user: sanjayss34
paper_authors._id: 64829bf6e4bbb1c2dd35da75
paper_authors.name: Medhini Narasimhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:35:16.781Z
paper_authors._id: 611ffea4e48a99edad639e8f
paper_authors.avatarUrl: /avatars/873ea85dbd5d79509faa3fedfcaa33d8.svg
paper_authors.fullname: Medhini Narasimhan
paper_authors.type: user
paper_authors.user: medhini
paper_authors._id: 64829bf6e4bbb1c2dd35da76
paper_authors.name: Kushal Khangaonkar
paper_authors._id: 64829bf6e4bbb1c2dd35da77
paper_authors.name: Kevin Yang
paper_authors._id: 64829bf6e4bbb1c2dd35da78
paper_authors.name: Arsha Nagrani
paper_authors._id: 64829bf6e4bbb1c2dd35da79
paper_authors.name: Cordelia Schmid
paper_authors._id: 64829bf6e4bbb1c2dd35da7a
paper_authors.name: Andy Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T09:12:29.862Z
paper_authors._id: 62d5f9d705461e9459d20b91
paper_authors.avatarUrl: /avatars/120917da2d8587674505163fb960e27d.svg
paper_authors.fullname: Andy Zeng
paper_authors.type: user
paper_authors.user: andyzeng
paper_authors._id: 64829bf6e4bbb1c2dd35da7b
paper_authors.name: Trevor Darrell
paper_authors._id: 64829bf6e4bbb1c2dd35da7c
paper_authors.name: Dan Klein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:36:14.695Z
paper_authors._id: 632be88b3690fb57e70e0bf1
paper_authors.avatarUrl: /avatars/74ff8f30b3662db2602495bdf493d397.svg
paper_authors.fullname: Dan Klein
paper_authors.type: user
paper_authors.user: danjklein
paper_title: Modular Visual Question Answering via Code Generation
paper_summary: We present a framework that formulates visual question answering as modular
  code generation. In contrast to prior work on modular approaches to VQA, our
  approach requires no additional training and relies on pre-trained language
  models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA
  examples used for in-context learning. The generated Python programs invoke and
  compose the outputs of the visual models using arithmetic and conditional
  logic. Our approach improves accuracy on the COVR dataset by at least 3% and on
  the GQA dataset by roughly 2% compared to the few-shot baseline that does not
  employ code generation.
paper_ai_keywords: visual question answering (VQA) modular code generation pre-trained language models (LMs) visual models image-caption pairs in-context learning Python programs arithmetic conditional logic COVR dataset GQA dataset few-shot baseline