AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal
  Understanding
Aligning visual features with language embeddings is a key challenge in
vision-language models (VLMs). The performance of such models hinges on having
a good connector that maps visual features generated by a vision encoder to a
shared embedding space with the LLM while preserving semantic similarity.
Existing connectors, such as multilayer perceptrons (MLPs), often produce
out-of-distribution or noisy inputs, leading to misalignment between the
modalities. In this work, we propose a novel vision-text alignment method,
AlignVLM, that maps visual features to a weighted average of LLM text
embeddings. Our approach leverages the linguistic priors encoded by the LLM to
ensure that visual features are mapped to regions of the space that the LLM can
effectively interpret. AlignVLM is particularly effective for document
understanding tasks, where scanned document images must be accurately mapped to
their textual content. Our extensive experiments show that AlignVLM achieves
state-of-the-art performance compared to prior alignment methods. We provide
further analysis demonstrating improved vision-text feature alignment and
robustness to noise.
67a236ba5f63ce00e8402d56
Ahmed Masry
claimed_verified
2025-02-04T16:54:31.623Z
63efd75a5c2ceb16fc6e98fc
https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
Ahmed Masry
user
ahmed-masry
67a236ba5f63ce00e8402d57
Juan A. Rodriguez
admin_assigned
2025-02-04T17:15:38.988Z
63507c18aef7e7f6cf476017
/avatars/183a74624b9daec613a57d405fa577bf.svg
Juan A. Rodriguez
user
joanrod
67a236ba5f63ce00e8402d58
Tianyu Zhang
admin_assigned
2025-02-04T17:15:45.137Z
6452d79149b6b9a2383b5775
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T28lP0kE7PZIGzJjhSpSx.jpeg
Tianyu Zhang
user
TianyuZhang
67a236ba5f63ce00e8402d59
Suyuchen Wang
claimed_verified
2025-02-04T16:54:29.585Z
62bb1e0f3ff437e49a3088e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/XuQKiRxyq9mncX5QrVLy7.jpeg
Suyuchen Wang
user
sheryc
67a236ba5f63ce00e8402d5a
Chao Wang
claimed_verified
2025-02-07T09:58:22.513Z
65826e30d73d6402f7ac515e
https://cdn-avatars.huggingface.co/v1/production/uploads/65826e30d73d6402f7ac515e/NjUQbyfMCWNjb5tVXTxKk.jpeg
Chao Wang
user
erikchwang
67a236ba5f63ce00e8402d5b
Aarash Feizi
admin_assigned
2025-02-04T17:33:25.590Z
6752203d99b478caa1e85a79
/avatars/29284c6cb11d45a640bf3871954007ed.svg
Aarash Feizi
user
feiziaarash
67a236ba5f63ce00e8402d5c
Akshay Kalkunte Suresh
67a236ba5f63ce00e8402d5d
Abhay Puri
admin_assigned
2025-02-04T17:33:52.015Z
65830af2a1707aa10effcc32
/avatars/0626454399b711fca7fb2b66fcecaca8.svg
Abhay Puri
user
abhaypuri
67a236ba5f63ce00e8402d5e
Xiangru Jian
admin_assigned
2025-02-04T17:33:57.470Z
66155491f1214e73d69074b5
/avatars/00572ab695a4188422e8ee38fc87680b.svg
Xiangru Jian
user
EdwardXJ
67a236ba5f63ce00e8402d5f
Pierre-André Noël
admin_assigned
2025-02-04T17:34:03.702Z
646cc16b94eb019a96e1fb2e
/avatars/31f168a6c1ec45eb0c784d9119c1b9bf.svg
Pierre-Andre Noel
user
PierreAndreNoel
67a236ba5f63ce00e8402d60
Sathwik Tejaswi Madhusudhan
admin_assigned
2025-02-04T17:34:10.257Z
63d3095c2727d7888cbb54e2
/avatars/51fd37f4216eec309cd439e56626d6ad.svg
Sathwik Tejaswi Madhusudhan
user
stm4
67a236ba5f63ce00e8402d61
Marco Pedersoli
admin_assigned
2025-02-04T17:34:16.472Z
64b829042fccad9f5ff20cc7
/avatars/92cca111ddd0db5d998615c2257a0894.svg
Marco Pedersoli
user
Marcopede
67a236ba5f63ce00e8402d62
Bang Liu
claimed_verified
2025-02-13T20:36:11.104Z
654a97282d2fcd6bf2851173
https://cdn-avatars.huggingface.co/v1/production/uploads/654a97282d2fcd6bf2851173/9zXf940gr4WNt4e-oOt4k.png
Bang Liu
user
Bang-UdeM-Mila
67a236ba5f63ce00e8402d63
Nicolas Chapados
admin_assigned
2025-02-04T17:34:23.259Z
631f54aa5ba8c026340b13cf
https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
Nicolas Chapados
user
nicolaschapados
67a236ba5f63ce00e8402d64
Yoshua Bengio
67a236ba5f63ce00e8402d65
Enamul Hoque
admin_assigned
2025-02-04T17:34:32.793Z
6706c57876c98ec236f2f090
/avatars/d45543b65b70f03a71dcd378a6ce931b.svg
Enamul Hoque
user
enamulhoque1
67a236ba5f63ce00e8402d66
Christopher Pal
67a236ba5f63ce00e8402d67
Issam H. Laradji
admin_assigned
2025-02-04T17:34:48.589Z
64062855692855e65ae31688
/avatars/e35d22a037b8b35422d3ee982f133076.svg
Issam Laradji
user
issamlaradji
67a236ba5f63ce00e8402d68
David Vazquez
admin_assigned
2025-02-04T17:35:01.844Z
646edffecb6ea6e6b6e1be4c
/avatars/8e1b0312c935ff1338c9fb74046fce02.svg
David Vazquez
user
DavidVazquez
67a236ba5f63ce00e8402d69
Perouz Taslakian
67a236ba5f63ce00e8402d6a
Spandana Gella
admin_assigned
2025-02-04T17:35:14.497Z
65ca8745d64d82c92fa7c71f
https://cdn-avatars.huggingface.co/v1/production/uploads/65ca8745d64d82c92fa7c71f/xayDYjs_LoAsa8BEgDlhb.jpeg
G
user
spandanagella
67a236ba5f63ce00e8402d6b
Sai Rajeswar
AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal
  Understanding
Aligning visual features with language embeddings is a key challenge in
vision-language models (VLMs). The performance of such models hinges on having
a good connector that maps visual features generated by a vision encoder to a
shared embedding space with the LLM while preserving semantic similarity.
Existing connectors, such as multilayer perceptrons (MLPs), often produce
out-of-distribution or noisy inputs, leading to misalignment between the
modalities. In this work, we propose a novel vision-text alignment method,
AlignVLM, that maps visual features to a weighted average of LLM text
embeddings. Our approach leverages the linguistic priors encoded by the LLM to
ensure that visual features are mapped to regions of the space that the LLM can
effectively interpret. AlignVLM is particularly effective for document
understanding tasks, where scanned document images must be accurately mapped to
their textual content. Our extensive experiments show that AlignVLM achieves
state-of-the-art performance compared to prior alignment methods. We provide
further analysis demonstrating improved vision-text feature alignment and
robustness to noise.
A new vision-text alignment method, AlignVLM, effectively maps visual features to LLM embeddings, improving performance in document understanding and robustness to noise.
vision-language models VLMs multilayer perceptrons MLPs AlignVLM LLM linguistic priors document understanding feature alignment