title: Jack of All Tasks, Master of Many: Designing General-purpose
    Coarse-to-Fine Vision-Language Model
summary: The ability of large language models (LLMs) to process visual inputs has
  given rise to general-purpose vision systems, unifying various vision-language
  (VL) tasks by instruction tuning. However, due to the enormous diversity in
  input-output formats in the vision domain, existing general-purpose models fail
  to successfully integrate segmentation and multi-image inputs with coarse-level
  tasks into a single framework. In this work, we introduce VistaLLM, a powerful
  visual system that addresses coarse- and fine-grained VL tasks over single and
  multiple input images using a unified framework. VistaLLM utilizes an
  instruction-guided image tokenizer that filters global embeddings using task
  descriptions to extract compressed and refined features from numerous images.
  Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to
  represent binary segmentation masks as sequences, significantly improving over
  previously used uniform sampling. To bolster the desired capability of
  VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning
  dataset with 6.8M samples. We also address the lack of multi-image grounding
  datasets by introducing a novel task, AttCoSeg (Attribute-level
  Co-Segmentation), which boosts the model's reasoning and grounding capability
  over multiple input images. Extensive experiments on a wide range of V- and VL
  tasks demonstrate the effectiveness of VistaLLM by achieving consistent
  state-of-the-art performance over strong baselines across all downstream tasks.
  Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.
paper_authors._id: 65825b92283fdb0c8de0deca
paper_authors.name: Shraman Pramanick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:27:22.997Z
paper_authors._id: 64aec10a9135f79b09117476
paper_authors.avatarUrl: /avatars/88a6bcfeee99c12496940575a2fc65e9.svg
paper_authors.fullname: Shraman Pramanick
paper_authors.type: user
paper_authors.user: Shraman
paper_authors._id: 65825b92283fdb0c8de0decb
paper_authors.name: Guangxing Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:28:33.248Z
paper_authors._id: 649a2c8fde0fb7f3f4969930
paper_authors.avatarUrl: /avatars/3767ad4fa2ffdf414a167b97bbc30352.svg
paper_authors.fullname: Guangxing Han
paper_authors.type: user
paper_authors.user: GuangxingHan
paper_authors._id: 65825b92283fdb0c8de0decc
paper_authors.name: Rui Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T08:19:38.770Z
paper_authors._id: 6407d236a7bc7c3865adcc3a
paper_authors.avatarUrl: /avatars/8b75cc07fd049acde5363d268c5b88d4.svg
paper_authors.fullname: Rui Hou
paper_authors.type: user
paper_authors.user: rayhou
paper_authors._id: 65825b92283fdb0c8de0decd
paper_authors.name: Sayan Nag
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:28:40.476Z
paper_authors._id: 61de7e35c201b96c5db3bec6
paper_authors.avatarUrl: /avatars/d69c9fcbeff60f6f1c442b2a1226307a.svg
paper_authors.fullname: Sayan Nag
paper_authors.type: user
paper_authors.user: nagsayan
paper_authors._id: 65825b92283fdb0c8de0dece
paper_authors.name: Ser-Nam Lim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-26T10:37:59.405Z
paper_authors._id: 65884498ce38d143c435d279
paper_authors.avatarUrl: /avatars/96a917c40680a79a37dcb8fa75014c21.svg
paper_authors.fullname: Ser Nam Lim
paper_authors.type: user
paper_authors.user: sernam
paper_authors._id: 65825b92283fdb0c8de0decf
paper_authors.name: Nicolas Ballas
paper_authors._id: 65825b92283fdb0c8de0ded0
paper_authors.name: Qifan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:29:33.421Z
paper_authors._id: 64b65c84c991daa61404abd7
paper_authors.avatarUrl: /avatars/3592aa8774ea4e2836099e0701b4b78d.svg
paper_authors.fullname: WangQifan
paper_authors.type: user
paper_authors.user: Bayees
paper_authors._id: 65825b92283fdb0c8de0ded1
paper_authors.name: Rama Chellappa
paper_authors._id: 65825b92283fdb0c8de0ded2
paper_authors.name: Amjad Almahairi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:30:13.811Z
paper_authors._id: 63ffacc4fe6383d50b2d2505
paper_authors.avatarUrl: /avatars/dd73c355000d0896bebced1c969804f2.svg
paper_authors.fullname: Amjad Almahairi
paper_authors.type: user
paper_authors.user: aalmah
paper_title: Jack of All Tasks, Master of Many: Designing General-purpose
    Coarse-to-Fine Vision-Language Model
paper_summary: The ability of large language models (LLMs) to process visual inputs has
  given rise to general-purpose vision systems, unifying various vision-language
  (VL) tasks by instruction tuning. However, due to the enormous diversity in
  input-output formats in the vision domain, existing general-purpose models fail
  to successfully integrate segmentation and multi-image inputs with coarse-level
  tasks into a single framework. In this work, we introduce VistaLLM, a powerful
  visual system that addresses coarse- and fine-grained VL tasks over single and
  multiple input images using a unified framework. VistaLLM utilizes an
  instruction-guided image tokenizer that filters global embeddings using task
  descriptions to extract compressed and refined features from numerous images.
  Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to
  represent binary segmentation masks as sequences, significantly improving over
  previously used uniform sampling. To bolster the desired capability of
  VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning
  dataset with 6.8M samples. We also address the lack of multi-image grounding
  datasets by introducing a novel task, AttCoSeg (Attribute-level
  Co-Segmentation), which boosts the model's reasoning and grounding capability
  over multiple input images. Extensive experiments on a wide range of V- and VL
  tasks demonstrate the effectiveness of VistaLLM by achieving consistent
  state-of-the-art performance over strong baselines across all downstream tasks.
  Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.
paper_ai_summary: VistaLLM is a unified framework for vision-language tasks that uses instruction-guided tokenization and adaptive sampling to process coarse- and fine-grained visual inputs with state-of-the-art results.
paper_ai_keywords: large language models vision-language tasks instruction tuning VistaLLM instruction-guided image tokenizer global embeddings gradient-aware adaptive sampling binary segmentation masks CoinIt multi-image grounding AttCoSeg Attribute-level Co-Segmentation