title: AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal
    Understanding
summary: Aligning visual features with language embeddings is a key challenge in
  vision-language models (VLMs). The performance of such models hinges on having
  a good connector that maps visual features generated by a vision encoder to a
  shared embedding space with the LLM while preserving semantic similarity.
  Existing connectors, such as multilayer perceptrons (MLPs), often produce
  out-of-distribution or noisy inputs, leading to misalignment between the
  modalities. In this work, we propose a novel vision-text alignment method,
  AlignVLM, that maps visual features to a weighted average of LLM text
  embeddings. Our approach leverages the linguistic priors encoded by the LLM to
  ensure that visual features are mapped to regions of the space that the LLM can
  effectively interpret. AlignVLM is particularly effective for document
  understanding tasks, where scanned document images must be accurately mapped to
  their textual content. Our extensive experiments show that AlignVLM achieves
  state-of-the-art performance compared to prior alignment methods. We provide
  further analysis demonstrating improved vision-text feature alignment and
  robustness to noise.
paper_authors._id: 67a236ba5f63ce00e8402d56
paper_authors.name: Ahmed Masry
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T16:54:31.623Z
paper_authors._id: 63efd75a5c2ceb16fc6e98fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
paper_authors.fullname: Ahmed Masry
paper_authors.type: user
paper_authors.user: ahmed-masry
paper_authors._id: 67a236ba5f63ce00e8402d57
paper_authors.name: Juan A. Rodriguez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:15:38.988Z
paper_authors._id: 63507c18aef7e7f6cf476017
paper_authors.avatarUrl: /avatars/183a74624b9daec613a57d405fa577bf.svg
paper_authors.fullname: Juan A. Rodriguez
paper_authors.type: user
paper_authors.user: joanrod
paper_authors._id: 67a236ba5f63ce00e8402d58
paper_authors.name: Tianyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:15:45.137Z
paper_authors._id: 6452d79149b6b9a2383b5775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T28lP0kE7PZIGzJjhSpSx.jpeg
paper_authors.fullname: Tianyu Zhang
paper_authors.type: user
paper_authors.user: TianyuZhang
paper_authors._id: 67a236ba5f63ce00e8402d59
paper_authors.name: Suyuchen Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T16:54:29.585Z
paper_authors._id: 62bb1e0f3ff437e49a3088e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/bcUQmH8tKfI6DIWH9IcYp.jpeg
paper_authors.fullname: Suyuchen Wang
paper_authors.type: user
paper_authors.user: sheryc
paper_authors._id: 67a236ba5f63ce00e8402d5a
paper_authors.name: Chao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T09:58:22.513Z
paper_authors._id: 65826e30d73d6402f7ac515e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65826e30d73d6402f7ac515e/NjUQbyfMCWNjb5tVXTxKk.jpeg
paper_authors.fullname: Chao Wang
paper_authors.type: user
paper_authors.user: erikchwang
paper_authors._id: 67a236ba5f63ce00e8402d5b
paper_authors.name: Aarash Feizi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:33:25.590Z
paper_authors._id: 6752203d99b478caa1e85a79
paper_authors.avatarUrl: /avatars/29284c6cb11d45a640bf3871954007ed.svg
paper_authors.fullname: Aarash Feizi
paper_authors.type: user
paper_authors.user: feiziaarash
paper_authors._id: 67a236ba5f63ce00e8402d5c
paper_authors.name: Akshay Kalkunte Suresh
paper_authors._id: 67a236ba5f63ce00e8402d5d
paper_authors.name: Abhay Puri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:33:52.015Z
paper_authors._id: 65830af2a1707aa10effcc32
paper_authors.avatarUrl: /avatars/0626454399b711fca7fb2b66fcecaca8.svg
paper_authors.fullname: Abhay Puri
paper_authors.type: user
paper_authors.user: abhaypuri
paper_authors._id: 67a236ba5f63ce00e8402d5e
paper_authors.name: Xiangru Jian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:33:57.470Z
paper_authors._id: 66155491f1214e73d69074b5
paper_authors.avatarUrl: /avatars/00572ab695a4188422e8ee38fc87680b.svg
paper_authors.fullname: Xiangru Jian
paper_authors.type: user
paper_authors.user: EdwardXJ
paper_authors._id: 67a236ba5f63ce00e8402d5f
paper_authors.name: Pierre-André Noël
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:34:03.702Z
paper_authors._id: 646cc16b94eb019a96e1fb2e
paper_authors.avatarUrl: /avatars/31f168a6c1ec45eb0c784d9119c1b9bf.svg
paper_authors.fullname: Pierre-Andre Noel
paper_authors.type: user
paper_authors.user: PierreAndreNoel
paper_authors._id: 67a236ba5f63ce00e8402d60
paper_authors.name: Sathwik Tejaswi Madhusudhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:34:10.257Z
paper_authors._id: 63d3095c2727d7888cbb54e2
paper_authors.avatarUrl: /avatars/51fd37f4216eec309cd439e56626d6ad.svg
paper_authors.fullname: Sathwik Tejaswi Madhusudhan
paper_authors.type: user
paper_authors.user: stm4
paper_authors._id: 67a236ba5f63ce00e8402d61
paper_authors.name: Marco Pedersoli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:34:16.472Z
paper_authors._id: 64b829042fccad9f5ff20cc7
paper_authors.avatarUrl: /avatars/92cca111ddd0db5d998615c2257a0894.svg
paper_authors.fullname: Marco Pedersoli
paper_authors.type: user
paper_authors.user: Marcopede
paper_authors._id: 67a236ba5f63ce00e8402d62
paper_authors.name: Bang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:36:11.104Z
paper_authors._id: 654a97282d2fcd6bf2851173
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654a97282d2fcd6bf2851173/9zXf940gr4WNt4e-oOt4k.png
paper_authors.fullname: Bang Liu
paper_authors.type: user
paper_authors.user: Bang-UdeM-Mila
paper_authors._id: 67a236ba5f63ce00e8402d63
paper_authors.name: Nicolas Chapados
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:34:23.259Z
paper_authors._id: 631f54aa5ba8c026340b13cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
paper_authors.fullname: Nicolas Chapados
paper_authors.type: user
paper_authors.user: nicolaschapados
paper_authors._id: 67a236ba5f63ce00e8402d64
paper_authors.name: Yoshua Bengio
paper_authors._id: 67a236ba5f63ce00e8402d65
paper_authors.name: Enamul Hoque
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:34:32.793Z
paper_authors._id: 6706c57876c98ec236f2f090
paper_authors.avatarUrl: /avatars/d45543b65b70f03a71dcd378a6ce931b.svg
paper_authors.fullname: Enamul Hoque
paper_authors.type: user
paper_authors.user: enamulhoque1
paper_authors._id: 67a236ba5f63ce00e8402d66
paper_authors.name: Christopher Pal
paper_authors._id: 67a236ba5f63ce00e8402d67
paper_authors.name: Issam H. Laradji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:34:48.589Z
paper_authors._id: 64062855692855e65ae31688
paper_authors.avatarUrl: /avatars/e35d22a037b8b35422d3ee982f133076.svg
paper_authors.fullname: Issam Laradji
paper_authors.type: user
paper_authors.user: issamlaradji
paper_authors._id: 67a236ba5f63ce00e8402d68
paper_authors.name: David Vazquez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:35:01.844Z
paper_authors._id: 646edffecb6ea6e6b6e1be4c
paper_authors.avatarUrl: /avatars/8e1b0312c935ff1338c9fb74046fce02.svg
paper_authors.fullname: David Vazquez
paper_authors.type: user
paper_authors.user: DavidVazquez
paper_authors._id: 67a236ba5f63ce00e8402d69
paper_authors.name: Perouz Taslakian
paper_authors._id: 67a236ba5f63ce00e8402d6a
paper_authors.name: Spandana Gella
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:35:14.497Z
paper_authors._id: 65ca8745d64d82c92fa7c71f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ca8745d64d82c92fa7c71f/xayDYjs_LoAsa8BEgDlhb.jpeg
paper_authors.fullname: G
paper_authors.type: user
paper_authors.user: spandanagella
paper_authors._id: 67a236ba5f63ce00e8402d6b
paper_authors.name: Sai Rajeswar
paper_title: AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal
    Understanding
paper_summary: Aligning visual features with language embeddings is a key challenge in
  vision-language models (VLMs). The performance of such models hinges on having
  a good connector that maps visual features generated by a vision encoder to a
  shared embedding space with the LLM while preserving semantic similarity.
  Existing connectors, such as multilayer perceptrons (MLPs), often produce
  out-of-distribution or noisy inputs, leading to misalignment between the
  modalities. In this work, we propose a novel vision-text alignment method,
  AlignVLM, that maps visual features to a weighted average of LLM text
  embeddings. Our approach leverages the linguistic priors encoded by the LLM to
  ensure that visual features are mapped to regions of the space that the LLM can
  effectively interpret. AlignVLM is particularly effective for document
  understanding tasks, where scanned document images must be accurately mapped to
  their textual content. Our extensive experiments show that AlignVLM achieves
  state-of-the-art performance compared to prior alignment methods. We provide
  further analysis demonstrating improved vision-text feature alignment and
  robustness to noise.
paper_ai_summary: A new vision-text alignment method, AlignVLM, effectively maps visual features to LLM embeddings, improving performance in document understanding and robustness to noise.
paper_ai_keywords: vision-language models VLMs multilayer perceptrons MLPs AlignVLM LLM linguistic priors document understanding feature alignment