title: Vector-ICL: In-context Learning with Continuous Vector Representations
summary: Large language models (LLMs) have shown remarkable in-context learning (ICL)
  capabilities on textual data. We explore whether these capabilities can be
  extended to continuous vectors from diverse domains, obtained from black-box
  pretrained encoders. By aligning input data with an LLM's embedding space
  through lightweight projectors, we observe that LLMs can effectively process
  and learn from these projected vectors, which we term Vector-ICL. In
  particular, we find that pretraining projectors with general language modeling
  objectives enables Vector-ICL, while task-specific finetuning further enhances
  performance. In our experiments across various tasks and modalities, including
  text reconstruction, numerical function regression, text classification,
  summarization, molecule captioning, time-series classification, graph
  classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL
  and domain-specific model or tuning. We further conduct analyses and case
  studies, indicating the potential of LLMs to process vector representations
  beyond traditional token-based paradigms.
paper_authors._id: 670961497a1bf6920b749308
paper_authors.name: Yufan Zhuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:16:02.444Z
paper_authors._id: 6438ccbb3b46237de3d052e8
paper_authors.avatarUrl: /avatars/baa624d417b0b905e82127dc66346478.svg
paper_authors.fullname: Yufan Zhuang
paper_authors.type: user
paper_authors.user: yzhuang
paper_authors._id: 670961497a1bf6920b749309
paper_authors.name: Chandan Singh
paper_authors._id: 670961497a1bf6920b74930a
paper_authors.name: Liyuan Liu
paper_authors._id: 670961497a1bf6920b74930b
paper_authors.name: Jingbo Shang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:51:32.459Z
paper_authors._id: 660655119e3555d648f6c6b5
paper_authors.avatarUrl: /avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
paper_authors.fullname: Jingbo Shang
paper_authors.type: user
paper_authors.user: shangjingbo
paper_authors._id: 670961497a1bf6920b74930c
paper_authors.name: Jianfeng Gao
paper_title: Vector-ICL: In-context Learning with Continuous Vector Representations
paper_summary: Large language models (LLMs) have shown remarkable in-context learning (ICL)
  capabilities on textual data. We explore whether these capabilities can be
  extended to continuous vectors from diverse domains, obtained from black-box
  pretrained encoders. By aligning input data with an LLM's embedding space
  through lightweight projectors, we observe that LLMs can effectively process
  and learn from these projected vectors, which we term Vector-ICL. In
  particular, we find that pretraining projectors with general language modeling
  objectives enables Vector-ICL, while task-specific finetuning further enhances
  performance. In our experiments across various tasks and modalities, including
  text reconstruction, numerical function regression, text classification,
  summarization, molecule captioning, time-series classification, graph
  classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL
  and domain-specific model or tuning. We further conduct analyses and case
  studies, indicating the potential of LLMs to process vector representations
  beyond traditional token-based paradigms.
paper_ai_summary: LLMs exhibit strong in-context learning capabilities with continuous vector inputs when fine-tuned with pretraining projectors, outperforming traditional methods across diverse tasks and modalities.
paper_ai_keywords: large language models in-context learning continuous vectors black-box pretrained encoders lightweight projectors embedding space general language modeling task-specific finetuning vector-ICL text reconstruction numerical function regression text classification summarization molecule captioning time-series classification graph classification fMRI decoding token-based paradigms