title: Measuring Progress in Fine-grained Vision-and-Language Understanding
summary: While pretraining on large-scale image-text data from the Web has facilitated
  rapid progress on many vision-and-language (V&L) tasks, recent work has
  demonstrated that pretrained models lack "fine-grained" understanding, such as
  the ability to recognise relationships, verbs, and numbers in images. This has
  resulted in an increased interest in the community to either develop new
  benchmarks or models for such capabilities. To better understand and quantify
  progress in this direction, we investigate four competitive V&L models on four
  fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al.,
  2022) consistently outperforms other baselines, and that modelling innovations
  can impact performance more than scaling Web data, which even degrades
  performance sometimes. Through a deeper investigation of X-VLM, we highlight
  the importance of both novel losses and rich data sources for learning
  fine-grained skills. Finally, we inspect training dynamics, and discover that
  for some tasks, performance peaks early in training or significantly
  fluctuates, never converging.
paper_authors._id: 6461cb755ea1bc9a9da94cda
paper_authors.name: Emanuele Bugliarello
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-15T15:28:30.074Z
paper_authors._id: 626a9284783d5891f45beb53
paper_authors.avatarUrl: /avatars/8737662babc5d42cafba6087ab33e716.svg
paper_authors.fullname: Emanuele Bugliarello
paper_authors.type: user
paper_authors.user: e-bug
paper_authors._id: 6461cb755ea1bc9a9da94cdb
paper_authors.name: Laurent Sartran
paper_authors._id: 6461cb755ea1bc9a9da94cdc
paper_authors.name: Aishwarya Agrawal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T12:12:50.228Z
paper_authors._id: 644fff2d17b6189cda58cb0e
paper_authors.avatarUrl: /avatars/03b367b9033c655129e48bd94bd245ee.svg
paper_authors.fullname: Aishwarya Agrawal
paper_authors.type: user
paper_authors.user: aagrawal
paper_authors._id: 6461cb755ea1bc9a9da94cdd
paper_authors.name: Lisa Anne Hendricks
paper_authors._id: 6461cb755ea1bc9a9da94cde
paper_authors.name: Aida Nematzadeh
paper_title: Measuring Progress in Fine-grained Vision-and-Language Understanding
paper_summary: While pretraining on large-scale image-text data from the Web has facilitated
  rapid progress on many vision-and-language (V&L) tasks, recent work has
  demonstrated that pretrained models lack "fine-grained" understanding, such as
  the ability to recognise relationships, verbs, and numbers in images. This has
  resulted in an increased interest in the community to either develop new
  benchmarks or models for such capabilities. To better understand and quantify
  progress in this direction, we investigate four competitive V&L models on four
  fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al.,
  2022) consistently outperforms other baselines, and that modelling innovations
  can impact performance more than scaling Web data, which even degrades
  performance sometimes. Through a deeper investigation of X-VLM, we highlight
  the importance of both novel losses and rich data sources for learning
  fine-grained skills. Finally, we inspect training dynamics, and discover that
  for some tasks, performance peaks early in training or significantly
  fluctuates, never converging.
paper_ai_keywords: vision-and-language (V&amp;L) tasks pretraining fine-grained understanding benchmarks X-VLM (Zeng et al., 2022) modelling innovations novel losses rich data sources training dynamics