title: Explanatory Instructions: Towards Unified Vision Tasks Understanding and
    Zero-shot Generalization
summary: Computer Vision (CV) has yet to fully achieve the zero-shot task
  generalization observed in Natural Language Processing (NLP), despite following
  many of the milestones established in NLP, such as large transformer models,
  extensive pre-training, and the auto-regression paradigm, among others. In this
  paper, we explore the idea that CV adopts discrete and terminological task
  definitions (\eg, ``image segmentation''), which may be a key barrier to
  zero-shot task generalization. Our hypothesis is that without truly
  understanding previously-seen tasks--due to these terminological
  definitions--deep models struggle to generalize to novel tasks. To verify this,
  we introduce Explanatory Instructions, which provide an intuitive way to define
  CV task objectives through detailed linguistic transformations from input
  images to outputs. We create a large-scale dataset comprising 12 million
  ``image input to explanatory instruction to output'' triplets, and train
  an auto-regressive-based vision-language model (AR-based VLM) that takes both
  images and explanatory instructions as input. By learning to follow these
  instructions, the AR-based VLM achieves instruction-level zero-shot
  capabilities for previously-seen tasks and demonstrates strong zero-shot
  generalization for unseen CV tasks. Code and dataset will be openly available
  on our GitHub repository.
paper_authors._id: 67721383d565d51e49e7a90f
paper_authors.name: Yang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:30:01.793Z
paper_authors._id: 66702b9cd8101e70bd8f70ec
paper_authors.avatarUrl: /avatars/2c20e4083ac314a4a42388b0ec4654e9.svg
paper_authors.fullname: sheny
paper_authors.type: user
paper_authors.user: axxkaya
paper_authors._id: 67721383d565d51e49e7a910
paper_authors.name: Xiu-Shen Wei
paper_authors._id: 67721383d565d51e49e7a911
paper_authors.name: Yifan Sun
paper_authors._id: 67721383d565d51e49e7a912
paper_authors.name: Yuxin Song
paper_authors._id: 67721383d565d51e49e7a913
paper_authors.name: Tao Yuan
paper_authors._id: 67721383d565d51e49e7a914
paper_authors.name: Jian Jin
paper_authors._id: 67721383d565d51e49e7a915
paper_authors.name: Heyang Xu
paper_authors._id: 67721383d565d51e49e7a916
paper_authors.name: Yazhou Yao
paper_authors._id: 67721383d565d51e49e7a917
paper_authors.name: Errui Ding
paper_title: Explanatory Instructions: Towards Unified Vision Tasks Understanding and
    Zero-shot Generalization
paper_summary: Computer Vision (CV) has yet to fully achieve the zero-shot task
  generalization observed in Natural Language Processing (NLP), despite following
  many of the milestones established in NLP, such as large transformer models,
  extensive pre-training, and the auto-regression paradigm, among others. In this
  paper, we explore the idea that CV adopts discrete and terminological task
  definitions (\eg, ``image segmentation''), which may be a key barrier to
  zero-shot task generalization. Our hypothesis is that without truly
  understanding previously-seen tasks--due to these terminological
  definitions--deep models struggle to generalize to novel tasks. To verify this,
  we introduce Explanatory Instructions, which provide an intuitive way to define
  CV task objectives through detailed linguistic transformations from input
  images to outputs. We create a large-scale dataset comprising 12 million
  ``image input to explanatory instruction to output'' triplets, and train
  an auto-regressive-based vision-language model (AR-based VLM) that takes both
  images and explanatory instructions as input. By learning to follow these
  instructions, the AR-based VLM achieves instruction-level zero-shot
  capabilities for previously-seen tasks and demonstrates strong zero-shot
  generalization for unseen CV tasks. Code and dataset will be openly available
  on our GitHub repository.
paper_ai_summary: An auto-regressive vision-language model trained with explanatory instructions achieves instruction-level zero-shot capabilities and strong zero-shot generalization in computer vision tasks.
paper_ai_keywords: transformer models pre-training auto-regression discrete task definitions vision-language model zero-shot task generalization image segmentation