title: Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue
    Learning
summary: Multimodal large language models (MLLMs), built on large-scale pre-trained
  vision towers and language models, have shown great capabilities in multimodal
  understanding. However, most existing MLLMs are trained on single-turn vision
  question-answering tasks, which do not accurately reflect real-world human
  conversations. In this paper, we introduce MMDiag, a multi-turn multimodal
  dialogue dataset. This dataset is collaboratively generated through
  deliberately designed rules and GPT assistance, featuring strong correlations
  between questions, between questions and images, and among different image
  regions; thus aligning more closely with real-world scenarios. MMDiag serves as
  a strong benchmark for multi-turn multimodal dialogue learning and brings more
  challenges to the grounding and reasoning capabilities of MLLMs. Further,
  inspired by human vision processing, we present DiagNote, an MLLM equipped with
  multimodal grounding and reasoning capabilities. DiagNote consists of two
  modules (Deliberate and Gaze) interacting with each other to perform
  Chain-of-Thought and annotations respectively, throughout multi-turn dialogues.
  We empirically demonstrate the advantages of DiagNote in both grounding and
  jointly processing and reasoning with vision and language information over
  existing MLLMs.
paper_authors._id: 67cfa814d212c9c5048845a0
paper_authors.name: Jiazheng Liu
paper_authors._id: 67cfa814d212c9c5048845a1
paper_authors.name: Sipeng Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T10:59:45.667Z
paper_authors._id: 64eac1f496f42afd627d439c
paper_authors.avatarUrl: /avatars/aa46265122b8a1170f57475494d7922e.svg
paper_authors.fullname: Sipeng Zheng
paper_authors.type: user
paper_authors.user: sipeng9527
paper_authors._id: 67cfa814d212c9c5048845a2
paper_authors.name: Börje F. Karlsson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:22:32.095Z
paper_authors._id: 61e52be53d6dbb1da842316a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
paper_authors.fullname: Börje Karlsson
paper_authors.type: user
paper_authors.user: tellarin
paper_authors._id: 67cfa814d212c9c5048845a3
paper_authors.name: Zongqing Lu
paper_title: Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue
    Learning
paper_summary: Multimodal large language models (MLLMs), built on large-scale pre-trained
  vision towers and language models, have shown great capabilities in multimodal
  understanding. However, most existing MLLMs are trained on single-turn vision
  question-answering tasks, which do not accurately reflect real-world human
  conversations. In this paper, we introduce MMDiag, a multi-turn multimodal
  dialogue dataset. This dataset is collaboratively generated through
  deliberately designed rules and GPT assistance, featuring strong correlations
  between questions, between questions and images, and among different image
  regions; thus aligning more closely with real-world scenarios. MMDiag serves as
  a strong benchmark for multi-turn multimodal dialogue learning and brings more
  challenges to the grounding and reasoning capabilities of MLLMs. Further,
  inspired by human vision processing, we present DiagNote, an MLLM equipped with
  multimodal grounding and reasoning capabilities. DiagNote consists of two
  modules (Deliberate and Gaze) interacting with each other to perform
  Chain-of-Thought and annotations respectively, throughout multi-turn dialogues.
  We empirically demonstrate the advantages of DiagNote in both grounding and
  jointly processing and reasoning with vision and language information over
  existing MLLMs.
paper_ai_keywords: multimodal large language models (MLLMs) vision towers multi-turn vision question-answering tasks multi-turn multimodal dialogue dataset (MMDiag) GPT assistant multimodal dialogue learning grounding reasoning capabilities Deliberate module Gaze module Chain-of-Thought