title: Link-Context Learning for Multimodal LLMs
summary: The ability to learn from context with novel concepts, and deliver
  appropriate responses are essential in human conversations. Despite current
  Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being
  trained on mega-scale datasets, recognizing unseen images or understanding
  novel concepts in a training-free manner remains a challenge. In-Context
  Learning (ICL) explores training-free few-shot learning, where models are
  encouraged to ``learn to learn" from limited tasks and generalize to unseen
  tasks. In this work, we propose link-context learning (LCL), which emphasizes
  "reasoning from cause and effect" to augment the learning capabilities of
  MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal
  relationship between the support set and the query set. By providing
  demonstrations with causal links, LCL guides the model to discern not only the
  analogy but also the underlying causal associations between data points, which
  empowers MLLMs to recognize unseen images and understand novel concepts more
  effectively. To facilitate the evaluation of this novel approach, we introduce
  the ISEKAI dataset, comprising exclusively of unseen generated image-label
  pairs designed for link-context learning. Extensive experiments show that our
  LCL-MLLM exhibits strong link-context learning capabilities to novel concepts
  over vanilla MLLMs. Code and data will be released at
  https://github.com/isekai-portal/Link-Context-Learning.
paper_authors._id: 64dc8a4698d50ccac684c74b
paper_authors.name: Yan Tai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-17T13:53:27.017Z
paper_authors._id: 647857b6c43296134a434fc6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Cs9p9xSXFlP3Y24JNV7Am.jpeg
paper_authors.fullname: taiyan
paper_authors.type: user
paper_authors.user: Amoik
paper_authors._id: 64dc8a4698d50ccac684c74c
paper_authors.name: Weichen Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-17T15:03:55.077Z
paper_authors._id: 6481764e8af4675862efb22e
paper_authors.avatarUrl: /avatars/fc2e076bc861693f598a528a068a696e.svg
paper_authors.fullname: weichenfan
paper_authors.type: user
paper_authors.user: weepiess2383
paper_authors._id: 64dc8a4698d50ccac684c74d
paper_authors.name: Zhao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:14:03.803Z
paper_authors._id: 62bc1adacaf01b9bec398547
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656494729797-noauth.png
paper_authors.fullname: Zhao Zhang
paper_authors.type: user
paper_authors.user: zbrl
paper_authors._id: 64dc8a4698d50ccac684c74e
paper_authors.name: Feng Zhu
paper_authors._id: 64dc8a4698d50ccac684c74f
paper_authors.name: Rui Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-16T14:41:22.403Z
paper_authors._id: 6405d7cca577649430bcceb9
paper_authors.avatarUrl: /avatars/befc09d3bc82685c8e205dcadfec13c0.svg
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: Robert0812
paper_authors._id: 64dc8a4698d50ccac684c750
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-16T14:38:40.872Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: Link-Context Learning for Multimodal LLMs
paper_summary: The ability to learn from context with novel concepts, and deliver
  appropriate responses are essential in human conversations. Despite current
  Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being
  trained on mega-scale datasets, recognizing unseen images or understanding
  novel concepts in a training-free manner remains a challenge. In-Context
  Learning (ICL) explores training-free few-shot learning, where models are
  encouraged to ``learn to learn" from limited tasks and generalize to unseen
  tasks. In this work, we propose link-context learning (LCL), which emphasizes
  "reasoning from cause and effect" to augment the learning capabilities of
  MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal
  relationship between the support set and the query set. By providing
  demonstrations with causal links, LCL guides the model to discern not only the
  analogy but also the underlying causal associations between data points, which
  empowers MLLMs to recognize unseen images and understand novel concepts more
  effectively. To facilitate the evaluation of this novel approach, we introduce
  the ISEKAI dataset, comprising exclusively of unseen generated image-label
  pairs designed for link-context learning. Extensive experiments show that our
  LCL-MLLM exhibits strong link-context learning capabilities to novel concepts
  over vanilla MLLMs. Code and data will be released at
  https://github.com/isekai-portal/Link-Context-Learning.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) Large Language Models (LLMs) In-Context Learning (ICL) few-shot learning link-context learning (LCL) causal relationship support set query set N ISEKAI dataset unseen generated image-label pairs link-context learning capabilities vanilla MLLMs