title: InternChat: Solving Vision-Centric Tasks by Interacting with Chatbots
    Beyond Language
summary: We present an interactive visual framework named InternChat, or iChat for
  short. The framework integrates chatbots that have planning and reasoning
  capabilities, such as ChatGPT, with non-verbal instructions like pointing
  movements that enable users to directly manipulate images or videos on the
  screen. Pointing (including gestures, cursors, etc.) movements can provide more
  flexibility and precision in performing vision-centric tasks that require
  fine-grained control, editing, and generation of visual content. The name
  InternChat stands for interaction, nonverbal, and chatbots. Different from
  existing interactive systems that rely on pure language, by incorporating
  pointing instructions, the proposed iChat significantly improves the efficiency
  of communication between users and chatbots, as well as the accuracy of
  chatbots in vision-centric tasks, especially in complicated visual scenarios
  where the number of objects is greater than 2. Additionally, in iChat, an
  auxiliary control mechanism is used to improve the control capability of LLM,
  and a large vision-language model termed Husky is fine-tuned for high-quality
  multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89% GPT-4 Quality).
  We hope this work can spark new ideas and directions for future interactive
  visual systems. Welcome to watch the code at
  https://github.com/OpenGVLab/InternChat.
paper_authors._id: 645aef997d8ec84b7b5e466f
paper_authors.name: Zhaoyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T13:44:41.363Z
paper_authors._id: 6432c6152bfb2b0ec7572479
paper_authors.avatarUrl: /avatars/7c2f908f511cdfa38a19e967d86f9c40.svg
paper_authors.fullname: Zhaoyang Liu
paper_authors.type: user
paper_authors.user: zyliu
paper_authors._id: 645aef997d8ec84b7b5e4670
paper_authors.name: Yinan He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:32:22.397Z
paper_authors._id: 62aafa49f29ff279b51f0182
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
paper_authors.fullname: yinanhe
paper_authors.type: user
paper_authors.user: ynhe
paper_authors._id: 645aef997d8ec84b7b5e4671
paper_authors.name: Wenhai Wang
paper_authors._id: 645aef997d8ec84b7b5e4672
paper_authors.name: Weiyun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:32:51.243Z
paper_authors._id: 619507e7b74b6c591f794340
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
paper_authors.fullname: Weiyun Wang
paper_authors.type: user
paper_authors.user: Weiyun1025
paper_authors._id: 645aef997d8ec84b7b5e4673
paper_authors.name: Yi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:49:33.545Z
paper_authors._id: 63119a987680dc699b2031df
paper_authors.avatarUrl: /avatars/db63ab73806ee9cb8aa01be82d9effdd.svg
paper_authors.fullname: Yi Wang
paper_authors.type: user
paper_authors.user: shepnerd
paper_authors._id: 645aef997d8ec84b7b5e4674
paper_authors.name: Shoufa Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:34:02.089Z
paper_authors._id: 6412a33900634c4fe9873652
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
paper_authors.fullname: Shoufa Chen
paper_authors.type: user
paper_authors.user: ShoufaChen
paper_authors._id: 645aef997d8ec84b7b5e4675
paper_authors.name: Qinglong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:34:46.454Z
paper_authors._id: 63db4d3b9f2687298a148b08
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675316592071-63db4d3b9f2687298a148b08.jpeg
paper_authors.fullname: Qinglong Zhang
paper_authors.type: user
paper_authors.user: wofmanaf
paper_authors._id: 645aef997d8ec84b7b5e4676
paper_authors.name: Yang Yang
paper_authors._id: 645aef997d8ec84b7b5e4677
paper_authors.name: Qingyun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:35:27.545Z
paper_authors._id: 637f100c1dbae09191088e4c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669271529782-noauth.jpeg
paper_authors.fullname: Qingyun Li
paper_authors.type: user
paper_authors.user: Qingyun
paper_authors._id: 645aef997d8ec84b7b5e4678
paper_authors.name: Jiashuo Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T07:10:40.478Z
paper_authors._id: 64b89a14cf14c2fabe96664c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
paper_authors.fullname: Jiashuo Yu
paper_authors.type: user
paper_authors.user: awojustin
paper_authors._id: 645aef997d8ec84b7b5e4679
paper_authors.name: Kunchang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:37:36.315Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 645aef997d8ec84b7b5e467a
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:51:48.003Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 645aef997d8ec84b7b5e467b
paper_authors.name: Xue Yang
paper_authors._id: 645aef997d8ec84b7b5e467c
paper_authors.name: Xizhou Zhu
paper_authors._id: 645aef997d8ec84b7b5e467d
paper_authors.name: Yali Wang
paper_authors._id: 645aef997d8ec84b7b5e467e
paper_authors.name: Limin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:38:51.807Z
paper_authors._id: 62c77f4352d8ae531f5511f9
paper_authors.avatarUrl: /avatars/50198ccb02ccd286975a4613fbabee28.svg
paper_authors.fullname: Limin Wang
paper_authors.type: user
paper_authors.user: lmwang
paper_authors._id: 645aef997d8ec84b7b5e467f
paper_authors.name: Ping Luo
paper_authors._id: 645aef997d8ec84b7b5e4680
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:39:39.552Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_authors._id: 645aef997d8ec84b7b5e4681
paper_authors.name: Yu Qiao
paper_title: InternChat: Solving Vision-Centric Tasks by Interacting with Chatbots
    Beyond Language
paper_summary: We present an interactive visual framework named InternChat, or iChat for
  short. The framework integrates chatbots that have planning and reasoning
  capabilities, such as ChatGPT, with non-verbal instructions like pointing
  movements that enable users to directly manipulate images or videos on the
  screen. Pointing (including gestures, cursors, etc.) movements can provide more
  flexibility and precision in performing vision-centric tasks that require
  fine-grained control, editing, and generation of visual content. The name
  InternChat stands for interaction, nonverbal, and chatbots. Different from
  existing interactive systems that rely on pure language, by incorporating
  pointing instructions, the proposed iChat significantly improves the efficiency
  of communication between users and chatbots, as well as the accuracy of
  chatbots in vision-centric tasks, especially in complicated visual scenarios
  where the number of objects is greater than 2. Additionally, in iChat, an
  auxiliary control mechanism is used to improve the control capability of LLM,
  and a large vision-language model termed Husky is fine-tuned for high-quality
  multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89% GPT-4 Quality).
  We hope this work can spark new ideas and directions for future interactive
  visual systems. Welcome to watch the code at
  https://github.com/OpenGVLab/InternChat.
paper_ai_keywords: iChat multimodal capabilities non-verbal instructions pointing movements vision-centric tasks fine-grained control editing generation of visual content chatbots vision-centric tasks visual scenarios objects auxiliary control mechanism large vision-language model Husky fine-tuning high-quality multi-modal dialogue ChatGPT-3.5-turbo GPT-4 Quality interactive visual systems