Task Vectors are Cross-Modal
We investigate the internal representations of vision-and-language models
(VLMs) and how they encode task representations. We consider tasks specified
through examples or instructions, using either text or image inputs.
Surprisingly, we find that conceptually similar tasks are mapped to similar
task vector representations, regardless of how they are specified. Our findings
suggest that to output answers, tokens in VLMs undergo three distinct phases:
input, task, and answer, a process which is consistent across different
modalities and specifications. The task vectors we identify in VLMs are general
enough to be derived in one modality (e.g., text) and transferred to another
(e.g., image). Additionally, we find that ensembling exemplar and instruction
based task vectors produce better task representations. Taken together, these
insights shed light on the underlying mechanisms of VLMs, particularly their
ability to represent tasks in a shared manner across different modalities and
task specifications. Project page:
https://task-vectors-are-cross-modal.github.io.
6721a30312c5d139aa076101
Grace Luo
claimed_verified
2024-10-30T09:52:16.601Z
630e5cc8c6b1d1bccb816f44
/avatars/ef1aa94fc793c2eb6f345eab98ae9cd9.svg
Grace Luo
user
g-luo
6721a30312c5d139aa076102
Trevor Darrell
6721a30312c5d139aa076103
Amir Bar
claimed_verified
2025-04-10T06:47:39.591Z
632bad473690fb57e70c9d42
/avatars/8c09bc614a7e62a7bd30c4cbb97d4122.svg
Amir Bar
user
amirbar1
Task Vectors are Cross-Modal
We investigate the internal representations of vision-and-language models
(VLMs) and how they encode task representations. We consider tasks specified
through examples or instructions, using either text or image inputs.
Surprisingly, we find that conceptually similar tasks are mapped to similar
task vector representations, regardless of how they are specified. Our findings
suggest that to output answers, tokens in VLMs undergo three distinct phases:
input, task, and answer, a process which is consistent across different
modalities and specifications. The task vectors we identify in VLMs are general
enough to be derived in one modality (e.g., text) and transferred to another
(e.g., image). Additionally, we find that ensembling exemplar and instruction
based task vectors produce better task representations. Taken together, these
insights shed light on the underlying mechanisms of VLMs, particularly their
ability to represent tasks in a shared manner across different modalities and
task specifications. Project page:
https://task-vectors-are-cross-modal.github.io.
Vision-and-language models encode conceptually similar tasks into similar vector representations across different modalities and specifications, undergoing an input-task-answer process that benefits from ensemble methods.
vision-and-language models VLMs task vector representations input task answer modalities task specifications exemplar instruction-based