title: GiT: Towards Generalist Vision Transformer through Universal Language
    Interface
summary: This paper proposes a simple, yet effective framework, called GiT,
  simultaneously applicable for various vision tasks only with a vanilla ViT.
  Motivated by the universality of the Multi-layer Transformer architecture (e.g,
  GPT) widely used in large language models (LLMs), we seek to broaden its scope
  to serve as a powerful vision foundation model (VFM). However, unlike language
  modeling, visual tasks typically require specific modules, such as bounding box
  heads for detection and pixel decoders for segmentation, greatly hindering the
  application of powerful multi-layer transformers in the vision domain. To solve
  this, we design a universal language interface that empowers the successful
  auto-regressive decoding to adeptly unify various visual tasks, from
  image-level understanding (e.g., captioning), over sparse perception (e.g.,
  detection), to dense prediction (e.g., segmentation). Based on the above
  designs, the entire model is composed solely of a ViT, without any specific
  additions, offering a remarkable architectural simplification. GiT is a
  multi-task visual model, jointly trained across five representative benchmarks
  without task-specific fine-tuning. Interestingly, our GiT builds a new
  benchmark in generalist performance, and fosters mutual enhancement across
  tasks, leading to significant improvements compared to isolated training. This
  reflects a similar impact observed in LLMs. Further enriching training with 27
  datasets, GiT achieves strong zero-shot results over various tasks. Due to its
  simple design, this paradigm holds promise for narrowing the architectural gap
  between vision and language. Code and models will be available at
  https://github.com/Haiyang-W/GiT.
paper_authors._id: 65f3b58e8081e7144e0224e6
paper_authors.name: Haiyang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T13:15:33.568Z
paper_authors._id: 65f43c3cc9940817caaf4434
paper_authors.avatarUrl: /avatars/ecec2856ba7a7d3421a2071a0a88800b.svg
paper_authors.fullname: Haiyang Wang
paper_authors.type: user
paper_authors.user: Haiyang-W
paper_authors._id: 65f3b58e8081e7144e0224e7
paper_authors.name: Hao Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T13:15:31.690Z
paper_authors._id: 6585493b53c37507639fe3ba
paper_authors.avatarUrl: /avatars/b7e71d4fa5ebb89a7ed6b2a8313687b5.svg
paper_authors.fullname: Hao Tang
paper_authors.type: user
paper_authors.user: kanashi6
paper_authors._id: 65f3b58e8081e7144e0224e8
paper_authors.name: Li Jiang
paper_authors._id: 65f3b58e8081e7144e0224e9
paper_authors.name: Shaoshuai Shi
paper_authors._id: 65f3b58e8081e7144e0224ea
paper_authors.name: Muhammad Ferjad Naeem
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T15:37:52.342Z
paper_authors._id: 67bf33a512368ec2fad4fe29
paper_authors.avatarUrl: /avatars/ea5c03744ec1c2bcc0e6c13efc8f7ddc.svg
paper_authors.fullname: Muhammad Ferjad Naeem
paper_authors.type: user
paper_authors.user: ferjad
paper_authors._id: 65f3b58e8081e7144e0224eb
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:24:42.244Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 65f3b58e8081e7144e0224ec
paper_authors.name: Bernt Schiele
paper_authors._id: 65f3b58e8081e7144e0224ed
paper_authors.name: Liwei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:24:59.495Z
paper_authors._id: 6497272b6401266eb6752b89
paper_authors.avatarUrl: /avatars/d94a011660f4cad5344756027840dc30.svg
paper_authors.fullname: Liwei Wang
paper_authors.type: user
paper_authors.user: asdfg80
paper_title: GiT: Towards Generalist Vision Transformer through Universal Language
    Interface
paper_summary: This paper proposes a simple, yet effective framework, called GiT,
  simultaneously applicable for various vision tasks only with a vanilla ViT.
  Motivated by the universality of the Multi-layer Transformer architecture (e.g,
  GPT) widely used in large language models (LLMs), we seek to broaden its scope
  to serve as a powerful vision foundation model (VFM). However, unlike language
  modeling, visual tasks typically require specific modules, such as bounding box
  heads for detection and pixel decoders for segmentation, greatly hindering the
  application of powerful multi-layer transformers in the vision domain. To solve
  this, we design a universal language interface that empowers the successful
  auto-regressive decoding to adeptly unify various visual tasks, from
  image-level understanding (e.g., captioning), over sparse perception (e.g.,
  detection), to dense prediction (e.g., segmentation). Based on the above
  designs, the entire model is composed solely of a ViT, without any specific
  additions, offering a remarkable architectural simplification. GiT is a
  multi-task visual model, jointly trained across five representative benchmarks
  without task-specific fine-tuning. Interestingly, our GiT builds a new
  benchmark in generalist performance, and fosters mutual enhancement across
  tasks, leading to significant improvements compared to isolated training. This
  reflects a similar impact observed in LLMs. Further enriching training with 27
  datasets, GiT achieves strong zero-shot results over various tasks. Due to its
  simple design, this paradigm holds promise for narrowing the architectural gap
  between vision and language. Code and models will be available at
  https://github.com/Haiyang-W/GiT.
paper_ai_summary: GiT is a versatile multi-task vision model using a vanilla ViT architecture that overcomes task-specific limitations and achieves competitive performance across various visual tasks through a universal language interface.
paper_ai_keywords: GiT ViT Multi-layer Transformer architecture bounding box heads pixel decoders visual tasks auto-regressive decoding image-level understanding sparse perception dense prediction generalist performance mutual enhancement zero-shot results