title: Task Vectors are Cross-Modal
summary: We investigate the internal representations of vision-and-language models
  (VLMs) and how they encode task representations. We consider tasks specified
  through examples or instructions, using either text or image inputs.
  Surprisingly, we find that conceptually similar tasks are mapped to similar
  task vector representations, regardless of how they are specified. Our findings
  suggest that to output answers, tokens in VLMs undergo three distinct phases:
  input, task, and answer, a process which is consistent across different
  modalities and specifications. The task vectors we identify in VLMs are general
  enough to be derived in one modality (e.g., text) and transferred to another
  (e.g., image). Additionally, we find that ensembling exemplar and instruction
  based task vectors produce better task representations. Taken together, these
  insights shed light on the underlying mechanisms of VLMs, particularly their
  ability to represent tasks in a shared manner across different modalities and
  task specifications. Project page:
  https://task-vectors-are-cross-modal.github.io.
paper_authors._id: 6721a30312c5d139aa076101
paper_authors.name: Grace Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:52:16.601Z
paper_authors._id: 630e5cc8c6b1d1bccb816f44
paper_authors.avatarUrl: /avatars/ef1aa94fc793c2eb6f345eab98ae9cd9.svg
paper_authors.fullname: Grace Luo
paper_authors.type: user
paper_authors.user: g-luo
paper_authors._id: 6721a30312c5d139aa076102
paper_authors.name: Trevor Darrell
paper_authors._id: 6721a30312c5d139aa076103
paper_authors.name: Amir Bar
paper_title: Task Vectors are Cross-Modal
paper_summary: We investigate the internal representations of vision-and-language models
  (VLMs) and how they encode task representations. We consider tasks specified
  through examples or instructions, using either text or image inputs.
  Surprisingly, we find that conceptually similar tasks are mapped to similar
  task vector representations, regardless of how they are specified. Our findings
  suggest that to output answers, tokens in VLMs undergo three distinct phases:
  input, task, and answer, a process which is consistent across different
  modalities and specifications. The task vectors we identify in VLMs are general
  enough to be derived in one modality (e.g., text) and transferred to another
  (e.g., image). Additionally, we find that ensembling exemplar and instruction
  based task vectors produce better task representations. Taken together, these
  insights shed light on the underlying mechanisms of VLMs, particularly their
  ability to represent tasks in a shared manner across different modalities and
  task specifications. Project page:
  https://task-vectors-are-cross-modal.github.io.
paper_ai_keywords: vision-and-language models (VLMs) task representations task vector representations input phase task phase answer phase exemplar-based task vectors instruction-based task vectors cross-modal