title: GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
summary: Instruction tuning large language model (LLM) on image-text pairs has
  achieved unprecedented vision-language multimodal abilities. However, their
  vision-language alignments are only built on image-level, the lack of
  region-level alignment limits their advancements to fine-grained multimodal
  understanding. In this paper, we propose instruction tuning on
  region-of-interest. The key design is to reformulate the bounding box as the
  format of spatial instruction. The interleaved sequences of visual features
  extracted by the spatial instruction and the language embedding are input to
  LLM, and trained on the transformed region-text data in instruction tuning
  format. Our region-level vision-language model, termed as GPT4RoI, brings brand
  new conversational and interactive experience beyond image-level understanding.
  (1) Controllability: Users can interact with our model by both language and
  spatial instructions to flexibly adjust the detail level of the question. (2)
  Capacities: Our model supports not only single-region spatial instruction but
  also multi-region. This unlocks more region-level multimodal capacities such as
  detailed region caption and complex region reasoning. (3) Composition: Any
  off-the-shelf object detector can be a spatial instruction provider so as to
  mine informative object attributes from our model, like color, shape, material,
  action, relation to other objects, etc. The code, data, and demo can be found
  at https://github.com/jshilong/GPT4RoI.
paper_authors._id: 64ab904189aa67e4a25736ee
paper_authors.name: Shilong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-10T10:05:59.341Z
paper_authors._id: 6424ffce46d202ad3d918a67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6424ffce46d202ad3d918a67/gmYmOA072fP_5cJLc9Qs4.jpeg
paper_authors.fullname: Shilong Zhang
paper_authors.type: user
paper_authors.user: shilongz
paper_authors._id: 64ab904189aa67e4a25736ef
paper_authors.name: Peize Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-10T10:06:18.103Z
paper_authors._id: 640dc9bf8512ec51d7f0ac1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
paper_authors.fullname: peizesun
paper_authors.type: user
paper_authors.user: peizesun
paper_authors._id: 64ab904189aa67e4a25736f0
paper_authors.name: Shoufa Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-10T10:06:38.369Z
paper_authors._id: 6412a33900634c4fe9873652
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
paper_authors.fullname: Shoufa Chen
paper_authors.type: user
paper_authors.user: ShoufaChen
paper_authors._id: 64ab904189aa67e4a25736f1
paper_authors.name: Min Xiao
paper_authors._id: 64ab904189aa67e4a25736f2
paper_authors.name: Wenqi Shao
paper_authors._id: 64ab904189aa67e4a25736f3
paper_authors.name: Wenwei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:36:17.647Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 64ab904189aa67e4a25736f4
paper_authors.name: Kai Chen
paper_authors._id: 64ab904189aa67e4a25736f5
paper_authors.name: Ping Luo
paper_title: GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
paper_summary: Instruction tuning large language model (LLM) on image-text pairs has
  achieved unprecedented vision-language multimodal abilities. However, their
  vision-language alignments are only built on image-level, the lack of
  region-level alignment limits their advancements to fine-grained multimodal
  understanding. In this paper, we propose instruction tuning on
  region-of-interest. The key design is to reformulate the bounding box as the
  format of spatial instruction. The interleaved sequences of visual features
  extracted by the spatial instruction and the language embedding are input to
  LLM, and trained on the transformed region-text data in instruction tuning
  format. Our region-level vision-language model, termed as GPT4RoI, brings brand
  new conversational and interactive experience beyond image-level understanding.
  (1) Controllability: Users can interact with our model by both language and
  spatial instructions to flexibly adjust the detail level of the question. (2)
  Capacities: Our model supports not only single-region spatial instruction but
  also multi-region. This unlocks more region-level multimodal capacities such as
  detailed region caption and complex region reasoning. (3) Composition: Any
  off-the-shelf object detector can be a spatial instruction provider so as to
  mine informative object attributes from our model, like color, shape, material,
  action, relation to other objects, etc. The code, data, and demo can be found
  at https://github.com/jshilong/GPT4RoI.
paper_ai_keywords: instruction tuning region-of-interest bounding box spatial instruction interleaved sequences visual features language embedding region-level vision-language model GPT4RoI controllability multi-region detailed region caption complex region reasoning composition object detector informative object attributes