title: Does Visual Pretraining Help End-to-End Reasoning?
summary: We aim to investigate whether end-to-end learning of visual reasoning can be
  achieved with general-purpose neural networks, with the help of visual
  pretraining. A positive result would refute the common belief that explicit
  visual abstraction (e.g. object detection) is essential for compositional
  generalization on visual reasoning, and confirm the feasibility of a neural
  network "generalist" to solve visual recognition and reasoning tasks. We
  propose a simple and general self-supervised framework which "compresses" each
  video frame into a small set of tokens with a transformer network, and
  reconstructs the remaining frames based on the compressed temporal context. To
  minimize the reconstruction loss, the network must learn a compact
  representation for each image, as well as capture temporal dynamics and object
  permanence from temporal context. We perform evaluation on two visual reasoning
  benchmarks, CATER and ACRE. We observe that pretraining is essential to achieve
  compositional generalization for end-to-end visual reasoning. Our proposed
  framework outperforms traditional supervised pretraining, including image
  classification and explicit object detection, by large margins.
paper_authors._id: 64b60d23d2a61d0b6adf4eaa
paper_authors.name: Chen Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-19T06:27:56.248Z
paper_authors._id: 64b6bac29bdb2adec64b1f86
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b6bac29bdb2adec64b1f86/516JDGISCKjtYvaYvprp4.jpeg
paper_authors.fullname: Chen Sun
paper_authors.type: user
paper_authors.user: jesu9
paper_authors._id: 64b60d23d2a61d0b6adf4eab
paper_authors.name: Calvin Luo
paper_authors._id: 64b60d23d2a61d0b6adf4eac
paper_authors.name: Xingyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:19:53.052Z
paper_authors._id: 629ee11846b4826be2d94676
paper_authors.avatarUrl: /avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
paper_authors.fullname: Xingyi Zhou
paper_authors.type: user
paper_authors.user: zhouxy
paper_authors._id: 64b60d23d2a61d0b6adf4ead
paper_authors.name: Anurag Arnab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:20:14.585Z
paper_authors._id: 63d7f0466ac3104e50c5311e
paper_authors.avatarUrl: /avatars/2421bfb7747664d1acefbac1c00c3475.svg
paper_authors.fullname: Anurag Arnab
paper_authors.type: user
paper_authors.user: anurag-arnab
paper_authors._id: 64b60d23d2a61d0b6adf4eae
paper_authors.name: Cordelia Schmid
paper_title: Does Visual Pretraining Help End-to-End Reasoning?
paper_summary: We aim to investigate whether end-to-end learning of visual reasoning can be
  achieved with general-purpose neural networks, with the help of visual
  pretraining. A positive result would refute the common belief that explicit
  visual abstraction (e.g. object detection) is essential for compositional
  generalization on visual reasoning, and confirm the feasibility of a neural
  network "generalist" to solve visual recognition and reasoning tasks. We
  propose a simple and general self-supervised framework which "compresses" each
  video frame into a small set of tokens with a transformer network, and
  reconstructs the remaining frames based on the compressed temporal context. To
  minimize the reconstruction loss, the network must learn a compact
  representation for each image, as well as capture temporal dynamics and object
  permanence from temporal context. We perform evaluation on two visual reasoning
  benchmarks, CATER and ACRE. We observe that pretraining is essential to achieve
  compositional generalization for end-to-end visual reasoning. Our proposed
  framework outperforms traditional supervised pretraining, including image
  classification and explicit object detection, by large margins.
paper_ai_keywords: visual pretraining visual reasoning end-to-end learning explicit visual abstraction object detection compositional generalization neural network self-supervised learning transformer network tokens temporal context reconstruction loss representation learning temporal dynamics object permanence visual recognition communication visual reasoning benchmarks Cater and ACRE pretraining traditional supervised pretraining image classification