SymDPO: Boosting In-Context Learning of Large Multimodal Models with
  Symbol Demonstration Direct Preference Optimization
As language models continue to scale, Large Language Models (LLMs) have
exhibited emerging capabilities in In-Context Learning (ICL), enabling them to
solve language tasks by prefixing a few in-context demonstrations (ICDs) as
context. Inspired by these advancements, researchers have extended these
techniques to develop Large Multimodal Models (LMMs) with ICL capabilities.
However, existing LMMs face a critical issue: they often fail to effectively
leverage the visual context in multimodal demonstrations and instead simply
follow textual patterns. This indicates that LMMs do not achieve effective
alignment between multimodal demonstrations and model outputs. To address this
problem, we propose Symbol Demonstration Direct Preference Optimization
(SymDPO). Specifically, SymDPO aims to break the traditional paradigm of
constructing multimodal demonstrations by using random symbols to replace text
answers within instances. This forces the model to carefully understand the
demonstration images and establish a relationship between the images and the
symbols to answer questions correctly. We validate the effectiveness of this
method on multiple benchmarks, demonstrating that with SymDPO, LMMs can more
effectively understand the multimodal context within examples and utilize this
knowledge to answer questions better.
673e8b6b72711222f0471639
Hongrui Jia
673e8b6b72711222f047163a
Chaoya Jiang
673e8b6b72711222f047163b
Haiyang Xu
claimed_verified
2024-11-21T08:09:42.565Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
673e8b6b72711222f047163c
Wei Ye
673e8b6b72711222f047163d
Mengfan Dong
673e8b6b72711222f047163e
Ming Yan
673e8b6b72711222f047163f
Ji Zhang
673e8b6b72711222f0471640
Fei Huang
673e8b6b72711222f0471641
Shikun Zhang
SymDPO: Boosting In-Context Learning of Large Multimodal Models with
  Symbol Demonstration Direct Preference Optimization
As language models continue to scale, Large Language Models (LLMs) have
exhibited emerging capabilities in In-Context Learning (ICL), enabling them to
solve language tasks by prefixing a few in-context demonstrations (ICDs) as
context. Inspired by these advancements, researchers have extended these
techniques to develop Large Multimodal Models (LMMs) with ICL capabilities.
However, existing LMMs face a critical issue: they often fail to effectively
leverage the visual context in multimodal demonstrations and instead simply
follow textual patterns. This indicates that LMMs do not achieve effective
alignment between multimodal demonstrations and model outputs. To address this
problem, we propose Symbol Demonstration Direct Preference Optimization
(SymDPO). Specifically, SymDPO aims to break the traditional paradigm of
constructing multimodal demonstrations by using random symbols to replace text
answers within instances. This forces the model to carefully understand the
demonstration images and establish a relationship between the images and the
symbols to answer questions correctly. We validate the effectiveness of this
method on multiple benchmarks, demonstrating that with SymDPO, LMMs can more
effectively understand the multimodal context within examples and utilize this
knowledge to answer questions better.
SymDPO optimizes preference for symbol demonstrations in large multimodal models, improving understanding and alignment in multimodal contexts.
Large Language Models In-Context Learning In-Context Demonstrations Large Multimodal Models Symbol Demonstration Direct Preference Optimization SymDPO