title: SymDPO: Boosting In-Context Learning of Large Multimodal Models with
    Symbol Demonstration Direct Preference Optimization
summary: As language models continue to scale, Large Language Models (LLMs) have
  exhibited emerging capabilities in In-Context Learning (ICL), enabling them to
  solve language tasks by prefixing a few in-context demonstrations (ICDs) as
  context. Inspired by these advancements, researchers have extended these
  techniques to develop Large Multimodal Models (LMMs) with ICL capabilities.
  However, existing LMMs face a critical issue: they often fail to effectively
  leverage the visual context in multimodal demonstrations and instead simply
  follow textual patterns. This indicates that LMMs do not achieve effective
  alignment between multimodal demonstrations and model outputs. To address this
  problem, we propose Symbol Demonstration Direct Preference Optimization
  (SymDPO). Specifically, SymDPO aims to break the traditional paradigm of
  constructing multimodal demonstrations by using random symbols to replace text
  answers within instances. This forces the model to carefully understand the
  demonstration images and establish a relationship between the images and the
  symbols to answer questions correctly. We validate the effectiveness of this
  method on multiple benchmarks, demonstrating that with SymDPO, LMMs can more
  effectively understand the multimodal context within examples and utilize this
  knowledge to answer questions better.
paper_authors._id: 673e8b6b72711222f0471639
paper_authors.name: Hongrui Jia
paper_authors._id: 673e8b6b72711222f047163a
paper_authors.name: Chaoya Jiang
paper_authors._id: 673e8b6b72711222f047163b
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T08:09:42.565Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 673e8b6b72711222f047163c
paper_authors.name: Wei Ye
paper_authors._id: 673e8b6b72711222f047163d
paper_authors.name: Mengfan Dong
paper_authors._id: 673e8b6b72711222f047163e
paper_authors.name: Ming Yan
paper_authors._id: 673e8b6b72711222f047163f
paper_authors.name: Ji Zhang
paper_authors._id: 673e8b6b72711222f0471640
paper_authors.name: Fei Huang
paper_authors._id: 673e8b6b72711222f0471641
paper_authors.name: Shikun Zhang
paper_title: SymDPO: Boosting In-Context Learning of Large Multimodal Models with
    Symbol Demonstration Direct Preference Optimization
paper_summary: As language models continue to scale, Large Language Models (LLMs) have
  exhibited emerging capabilities in In-Context Learning (ICL), enabling them to
  solve language tasks by prefixing a few in-context demonstrations (ICDs) as
  context. Inspired by these advancements, researchers have extended these
  techniques to develop Large Multimodal Models (LMMs) with ICL capabilities.
  However, existing LMMs face a critical issue: they often fail to effectively
  leverage the visual context in multimodal demonstrations and instead simply
  follow textual patterns. This indicates that LMMs do not achieve effective
  alignment between multimodal demonstrations and model outputs. To address this
  problem, we propose Symbol Demonstration Direct Preference Optimization
  (SymDPO). Specifically, SymDPO aims to break the traditional paradigm of
  constructing multimodal demonstrations by using random symbols to replace text
  answers within instances. This forces the model to carefully understand the
  demonstration images and establish a relationship between the images and the
  symbols to answer questions correctly. We validate the effectiveness of this
  method on multiple benchmarks, demonstrating that with SymDPO, LMMs can more
  effectively understand the multimodal context within examples and utilize this
  knowledge to answer questions better.
paper_ai_keywords: Large Language Models (LLMs) In-Context Learning (ICL) In-Context Demonstrations (ICDs) Large Multimodal Models (LMMs) Symbol Demonstration Direct Preference Optimization (SymDPO)