Explanatory Instructions: Towards Unified Vision Tasks Understanding and
  Zero-shot Generalization
Computer Vision (CV) has yet to fully achieve the zero-shot task
generalization observed in Natural Language Processing (NLP), despite following
many of the milestones established in NLP, such as large transformer models,
extensive pre-training, and the auto-regression paradigm, among others. In this
paper, we explore the idea that CV adopts discrete and terminological task
definitions (\eg, ``image segmentation''), which may be a key barrier to
zero-shot task generalization. Our hypothesis is that without truly
understanding previously-seen tasks--due to these terminological
definitions--deep models struggle to generalize to novel tasks. To verify this,
we introduce Explanatory Instructions, which provide an intuitive way to define
CV task objectives through detailed linguistic transformations from input
images to outputs. We create a large-scale dataset comprising 12 million
``image input to explanatory instruction to output'' triplets, and train
an auto-regressive-based vision-language model (AR-based VLM) that takes both
images and explanatory instructions as input. By learning to follow these
instructions, the AR-based VLM achieves instruction-level zero-shot
capabilities for previously-seen tasks and demonstrates strong zero-shot
generalization for unseen CV tasks. Code and dataset will be openly available
on our GitHub repository.
67721383d565d51e49e7a90f
Yang Shen
claimed_verified
2024-12-30T19:30:01.793Z
66702b9cd8101e70bd8f70ec
/avatars/2c20e4083ac314a4a42388b0ec4654e9.svg
sheny
user
axxkaya
67721383d565d51e49e7a910
Xiu-Shen Wei
67721383d565d51e49e7a911
Yifan Sun
67721383d565d51e49e7a912
Yuxin Song
67721383d565d51e49e7a913
Tao Yuan
67721383d565d51e49e7a914
Jian Jin
67721383d565d51e49e7a915
Heyang Xu
67721383d565d51e49e7a916
Yazhou Yao
67721383d565d51e49e7a917
Errui Ding
Explanatory Instructions: Towards Unified Vision Tasks Understanding and
  Zero-shot Generalization
Computer Vision (CV) has yet to fully achieve the zero-shot task
generalization observed in Natural Language Processing (NLP), despite following
many of the milestones established in NLP, such as large transformer models,
extensive pre-training, and the auto-regression paradigm, among others. In this
paper, we explore the idea that CV adopts discrete and terminological task
definitions (\eg, ``image segmentation''), which may be a key barrier to
zero-shot task generalization. Our hypothesis is that without truly
understanding previously-seen tasks--due to these terminological
definitions--deep models struggle to generalize to novel tasks. To verify this,
we introduce Explanatory Instructions, which provide an intuitive way to define
CV task objectives through detailed linguistic transformations from input
images to outputs. We create a large-scale dataset comprising 12 million
``image input to explanatory instruction to output'' triplets, and train
an auto-regressive-based vision-language model (AR-based VLM) that takes both
images and explanatory instructions as input. By learning to follow these
instructions, the AR-based VLM achieves instruction-level zero-shot
capabilities for previously-seen tasks and demonstrates strong zero-shot
generalization for unseen CV tasks. Code and dataset will be openly available
on our GitHub repository.
An auto-regressive vision-language model trained with explanatory instructions achieves instruction-level zero-shot capabilities and strong zero-shot generalization in computer vision tasks.
transformer models pre-training auto-regression discrete task definitions vision-language model zero-shot task generalization image segmentation