title: NExT-Chat: An LMM for Chat, Detection and Segmentation
summary: The development of large language models (LLMs) has greatly advanced the
  field of multimodal understanding, leading to the emergence of large multimodal
  models (LMMs). In order to enhance the level of visual comprehension, recent
  studies have equipped LMMs with region-level understanding capabilities by
  representing object bounding box coordinates as a series of text sequences
  (pixel2seq). In this paper, we introduce a novel paradigm for object location
  modeling called pixel2emb method, where we ask the LMM to output the location
  embeddings and then decoded by different decoders. This paradigm allows for
  different location formats (such as bounding boxes and masks) to be used in
  multimodal conversations Furthermore, this kind of embedding based location
  modeling enables the utilization of existing practices in localization tasks,
  such as detection and segmentation. In scenarios with limited resources, our
  pixel2emb demonstrates superior performance compared to existing
  state-of-the-art (SOTA) approaches in both the location input and output tasks
  under fair comparison. Leveraging the proposed pixel2emb method, we train an
  LMM named NExT-Chat and demonstrate its capability of handling multiple tasks
  like visual grounding, region caption, and grounded reasoning.
paper_authors._id: 654c49333e8861928003c7ab
paper_authors.name: Ao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-09T09:48:38.110Z
paper_authors._id: 649a9133225ce03f1c82e0b2
paper_authors.avatarUrl: /avatars/f3702a46a0a310d0292dc690794e7cc9.svg
paper_authors.fullname: Ao Zhang
paper_authors.type: user
paper_authors.user: AoZhang
paper_authors._id: 654c49333e8861928003c7ac
paper_authors.name: Liming Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-09T10:14:56.750Z
paper_authors._id: 629867eb52ad3199b9fe964c
paper_authors.avatarUrl: /avatars/54a79d6b9affec95f1244f45ff714137.svg
paper_authors.fullname: Liming
paper_authors.type: user
paper_authors.user: liming712
paper_authors._id: 654c49333e8861928003c7ad
paper_authors.name: Chen-Wei Xie
paper_authors._id: 654c49333e8861928003c7ae
paper_authors.name: Yun Zheng
paper_authors._id: 654c49333e8861928003c7af
paper_authors.name: Wei Ji
paper_authors._id: 654c49333e8861928003c7b0
paper_authors.name: Tat-Seng Chua
paper_title: NExT-Chat: An LMM for Chat, Detection and Segmentation
paper_summary: The development of large language models (LLMs) has greatly advanced the
  field of multimodal understanding, leading to the emergence of large multimodal
  models (LMMs). In order to enhance the level of visual comprehension, recent
  studies have equipped LMMs with region-level understanding capabilities by
  representing object bounding box coordinates as a series of text sequences
  (pixel2seq). In this paper, we introduce a novel paradigm for object location
  modeling called pixel2emb method, where we ask the LMM to output the location
  embeddings and then decoded by different decoders. This paradigm allows for
  different location formats (such as bounding boxes and masks) to be used in
  multimodal conversations Furthermore, this kind of embedding based location
  modeling enables the utilization of existing practices in localization tasks,
  such as detection and segmentation. In scenarios with limited resources, our
  pixel2emb demonstrates superior performance compared to existing
  state-of-the-art (SOTA) approaches in both the location input and output tasks
  under fair comparison. Leveraging the proposed pixel2emb method, we train an
  LMM named NExT-Chat and demonstrate its capability of handling multiple tasks
  like visual grounding, region caption, and grounded reasoning.
paper_ai_summary: The pixel2emb method enables large multimodal models to effectively handle object location tasks by outputting location embeddings, leading to superior performance in multimodal conversations and various visual processing tasks.
paper_ai_keywords: large language models (LLMs) large multimodal models (LMMs) region-level understanding bounding box coordinates pixel2seq pixel2emb location embeddings multimodal conversations object location modeling detection segmentation visual grounding region caption grounded reasoning