title: An Introduction to Vision-Language Modeling
summary: Following the recent popularity of Large Language Models (LLMs), several
  attempts have been made to extend them to the visual domain. From having a
  visual assistant that could guide us through unfamiliar environments to
  generative models that produce images using only a high-level text description,
  the vision-language model (VLM) applications will significantly impact our
  relationship with technology. However, there are many challenges that need to
  be addressed to improve the reliability of those models. While language is
  discrete, vision evolves in a much higher dimensional space in which concepts
  cannot always be easily discretized. To better understand the mechanics behind
  mapping vision to language, we present this introduction to VLMs which we hope
  will help anyone who would like to enter the field. First, we introduce what
  VLMs are, how they work, and how to train them. Then, we present and discuss
  approaches to evaluate VLMs. Although this work primarily focuses on mapping
  images to language, we also discuss extending VLMs to videos.
paper_authors._id: 66554c1f91494de1ddb4d25e
paper_authors.name: Florian Bordes
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-28T10:51:48.271Z
paper_authors._id: 63166336894404e2506b8811
paper_authors.avatarUrl: /avatars/ec51a1dd86511127cf83afd4bf7c0f52.svg
paper_authors.fullname: Florian Bordes
paper_authors.type: user
paper_authors.user: Fbordes
paper_authors._id: 66554c1f91494de1ddb4d25f
paper_authors.name: Richard Yuanzhe Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:23:32.096Z
paper_authors._id: 64d50fa59a6a7ae984d90852
paper_authors.avatarUrl: /avatars/26b78c151e707e3f9a9516c877d18ff7.svg
paper_authors.fullname: Richard Pang
paper_authors.type: user
paper_authors.user: yzpang
paper_authors._id: 66554c1f91494de1ddb4d260
paper_authors.name: Anurag Ajay
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:23:44.241Z
paper_authors._id: 64a7345b23622f7f188cea55
paper_authors.avatarUrl: /avatars/069162d6d85bb0d9e810c24a867aff7b.svg
paper_authors.fullname: Anurag Ajay
paper_authors.type: user
paper_authors.user: anuragajay
paper_authors._id: 66554c1f91494de1ddb4d261
paper_authors.name: Alexander C. Li
paper_authors._id: 66554c1f91494de1ddb4d262
paper_authors.name: Adrien Bardes
paper_authors._id: 66554c1f91494de1ddb4d263
paper_authors.name: Suzanne Petryk
paper_authors._id: 66554c1f91494de1ddb4d264
paper_authors.name: Oscar Mañas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:24:15.589Z
paper_authors._id: 6355d5c460c1b72f62693983
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6355d5c460c1b72f62693983/RA4FJCgPyqMXgTaSvBUGq.jpeg
paper_authors.fullname: Oscar Mañas
paper_authors.type: user
paper_authors.user: oscmansan
paper_authors._id: 66554c1f91494de1ddb4d265
paper_authors.name: Zhiqiu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:25:06.934Z
paper_authors._id: 64c170190bfb901b04399295
paper_authors.avatarUrl: /avatars/c30ce7566ae3497ddc989ec8918d37cc.svg
paper_authors.fullname: Zhiqiu Lin
paper_authors.type: user
paper_authors.user: zhiqiulin
paper_authors._id: 66554c1f91494de1ddb4d266
paper_authors.name: Anas Mahmoud
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:25:16.518Z
paper_authors._id: 64a588d26c0223510c6047a9
paper_authors.avatarUrl: /avatars/6cc422be5cd2f0f78c0b0cc9c3743e02.svg
paper_authors.fullname: Anas Mahmoud
paper_authors.type: user
paper_authors.user: nasmahmoud
paper_authors._id: 66554c1f91494de1ddb4d267
paper_authors.name: Bargav Jayaraman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:25:22.476Z
paper_authors._id: 661ddb6c554eb4b8b86f872a
paper_authors.avatarUrl: /avatars/6829d1466ea28a363ca7c812401fe332.svg
paper_authors.fullname: Bargav Jayaraman
paper_authors.type: user
paper_authors.user: bargavj
paper_authors._id: 66554c1f91494de1ddb4d268
paper_authors.name: Mark Ibrahim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T15:20:11.701Z
paper_authors._id: 64c17345e82e55936cf971bc
paper_authors.avatarUrl: /avatars/b39c5637e00291410620afb7c770d587.svg
paper_authors.fullname: Mark Ibrahim
paper_authors.type: user
paper_authors.user: marksibrahim
paper_authors._id: 66554c1f91494de1ddb4d269
paper_authors.name: Melissa Hall
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:45.585Z
paper_authors._id: 6697bfb4acbced03ebd133f8
paper_authors.avatarUrl: /avatars/a7d41367c6f7974303feb5af2a387ce4.svg
paper_authors.fullname: Melissa Hall
paper_authors.type: user
paper_authors.user: melissah14
paper_authors._id: 66554c1f91494de1ddb4d26a
paper_authors.name: Yunyang Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:18:54.764Z
paper_authors._id: 65304b62e7535baecd85d080
paper_authors.avatarUrl: /avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
paper_authors.fullname: Yunyang Xiong
paper_authors.type: user
paper_authors.user: yunyangx
paper_authors._id: 66554c1f91494de1ddb4d26b
paper_authors.name: Jonathan Lebensold
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:25:36.765Z
paper_authors._id: 6515a093c33a8b191953902c
paper_authors.avatarUrl: /avatars/871ae82d37c63bc5e2dd15726b845d41.svg
paper_authors.fullname: Jonathan Lebensold
paper_authors.type: user
paper_authors.user: jlebensold
paper_authors._id: 66554c1f91494de1ddb4d26c
paper_authors.name: Candace Ross
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:25:52.567Z
paper_authors._id: 628709d57cc3b7bc1b95fcf0
paper_authors.avatarUrl: /avatars/98e0d9695527125a0409d1098cdceaa4.svg
paper_authors.fullname: Candace Ross
paper_authors.type: user
paper_authors.user: candaceross
paper_authors._id: 66554c1f91494de1ddb4d26d
paper_authors.name: Srihari Jayakumar
paper_authors._id: 66554c1f91494de1ddb4d26e
paper_authors.name: Chuan Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:26:10.065Z
paper_authors._id: 64c933cfdc59e43a6109b065
paper_authors.avatarUrl: /avatars/b7f64fae8df8704c4911065759abfcd3.svg
paper_authors.fullname: Chuan Guo
paper_authors.type: user
paper_authors.user: chuanguo
paper_authors._id: 66554c1f91494de1ddb4d26f
paper_authors.name: Diane Bouchacourt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:26:16.384Z
paper_authors._id: 641e01d13bae5a776368311b
paper_authors.avatarUrl: /avatars/1e983fb59f475b3252e201f44eec4657.svg
paper_authors.fullname: Diane Bouchacourt
paper_authors.type: user
paper_authors.user: DianeB
paper_authors._id: 66554c1f91494de1ddb4d270
paper_authors.name: Haider Al-Tahan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:26:26.010Z
paper_authors._id: 63654c5edaa65e06f90cc5ee
paper_authors.avatarUrl: /avatars/11fd0c361531f97b16b91629d0083b92.svg
paper_authors.fullname: haider
paper_authors.type: user
paper_authors.user: haideraltahan
paper_authors._id: 66554c1f91494de1ddb4d271
paper_authors.name: Karthik Padthe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:26:32.153Z
paper_authors._id: 64c01530bd4b58768d8b46b5
paper_authors.avatarUrl: /avatars/0f3c0cdab16e3078b37569d73ca03792.svg
paper_authors.fullname: Karthik padthe
paper_authors.type: user
paper_authors.user: Karthikpadthe
paper_authors._id: 66554c1f91494de1ddb4d272
paper_authors.name: Vasu Sharma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:26:54.171Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 66554c1f91494de1ddb4d273
paper_authors.name: Hu Xu
paper_authors._id: 66554c1f91494de1ddb4d274
paper_authors.name: Xiaoqing Ellen Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:27:42.421Z
paper_authors._id: 6345ab8ffe134dfd7a0d1059
paper_authors.avatarUrl: /avatars/159a364ac4f8a2ed001346d4cbb7f2e3.svg
paper_authors.fullname: Ellen Tan
paper_authors.type: user
paper_authors.user: ellenxtan
paper_authors._id: 66554c1f91494de1ddb4d275
paper_authors.name: Megan Richards
paper_authors._id: 66554c1f91494de1ddb4d276
paper_authors.name: Samuel Lavoie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:28:27.013Z
paper_authors._id: 64fdd1509132c7f62a49b16b
paper_authors.avatarUrl: /avatars/b3da596b4f622e0acb0c1d0e2ca8643e.svg
paper_authors.fullname: Samuel Lavoie
paper_authors.type: user
paper_authors.user: lavoiems
paper_authors._id: 66554c1f91494de1ddb4d277
paper_authors.name: Pietro Astolfi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:28:34.950Z
paper_authors._id: 646388dfd4d34b01f45d87c3
paper_authors.avatarUrl: /avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
paper_authors.fullname: Pietro Astolfi
paper_authors.type: user
paper_authors.user: pietroastolfi
paper_authors._id: 66554c1f91494de1ddb4d278
paper_authors.name: Reyhane Askari Hemmat
paper_authors._id: 66554c1f91494de1ddb4d279
paper_authors.name: Jun Chen
paper_authors._id: 66554c1f91494de1ddb4d27a
paper_authors.name: Kushal Tirumala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:29:57.624Z
paper_authors._id: 6429dac775bcc24c5e579f5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GI73-WJ5WjdB-KUUNGkdv.jpeg
paper_authors.fullname: Kushal Tirumala
paper_authors.type: user
paper_authors.user: kushaltirumala
paper_authors._id: 66554c1f91494de1ddb4d27b
paper_authors.name: Rim Assouel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:30:05.857Z
paper_authors._id: 65f49a8c9a664d06a3d7d10e
paper_authors.avatarUrl: /avatars/46a8b6f4f75fe9c375794398cdbc7a42.svg
paper_authors.fullname: Rim Assouel
paper_authors.type: user
paper_authors.user: rassouel
paper_authors._id: 66554c1f91494de1ddb4d27c
paper_authors.name: Mazda Moayeri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:58:59.637Z
paper_authors._id: 63449874ee1504dbcd59af3d
paper_authors.avatarUrl: /avatars/57a805d82d16de9544c98585bd7a3e55.svg
paper_authors.fullname: MazdaM
paper_authors.type: user
paper_authors.user: mmoayeri
paper_authors._id: 66554c1f91494de1ddb4d27d
paper_authors.name: Arjang Talattof
paper_authors._id: 66554c1f91494de1ddb4d27e
paper_authors.name: Kamalika Chaudhuri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:30.438Z
paper_authors._id: 674fd6355d220432321ae8da
paper_authors.avatarUrl: /avatars/69839b5cb44a159c711a4dd6d8046477.svg
paper_authors.fullname: Kamalika Chaudhuri
paper_authors.type: user
paper_authors.user: kamalikac
paper_authors._id: 66554c1f91494de1ddb4d27f
paper_authors.name: Zechun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:30:27.107Z
paper_authors._id: 660f893bae89429c07a32cdb
paper_authors.avatarUrl: /avatars/27442b1dab58114cfe10220c040c1156.svg
paper_authors.fullname: Zechun Liu
paper_authors.type: user
paper_authors.user: zechunliu
paper_authors._id: 66554c1f91494de1ddb4d280
paper_authors.name: Xilun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:30:44.156Z
paper_authors._id: 62ffccd18813dd8628172c6c
paper_authors.avatarUrl: /avatars/62741be5967a74758a2b7914aa842ec8.svg
paper_authors.fullname: Xilun Chen
paper_authors.type: user
paper_authors.user: ccsasuke
paper_authors._id: 66554c1f91494de1ddb4d281
paper_authors.name: Quentin Garrido
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:30:51.045Z
paper_authors._id: 63049022412a1b9d381b9dcb
paper_authors.avatarUrl: /avatars/cac2a540fda40c5ae558968c50385f46.svg
paper_authors.fullname: Quentin Garrido
paper_authors.type: user
paper_authors.user: garridoq
paper_authors._id: 66554c1f91494de1ddb4d282
paper_authors.name: Karen Ullrich
paper_authors._id: 66554c1f91494de1ddb4d283
paper_authors.name: Aishwarya Agrawal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:31:03.756Z
paper_authors._id: 644fff2d17b6189cda58cb0e
paper_authors.avatarUrl: /avatars/03b367b9033c655129e48bd94bd245ee.svg
paper_authors.fullname: Aishwarya Agrawal
paper_authors.type: user
paper_authors.user: aagrawal
paper_authors._id: 66554c1f91494de1ddb4d284
paper_authors.name: Kate Saenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:31:12.358Z
paper_authors._id: 6272e87b72307af646a5372c
paper_authors.avatarUrl: /avatars/258f3977ae88d747892dcf83b2c56c2e.svg
paper_authors.fullname: Kate Saenko
paper_authors.type: user
paper_authors.user: KateSaenko
paper_authors._id: 66554c1f91494de1ddb4d285
paper_authors.name: Asli Celikyilmaz
paper_authors._id: 66554c1f91494de1ddb4d286
paper_authors.name: Vikas Chandra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:31:35.359Z
paper_authors._id: 6566681f7b5ed0735812af32
paper_authors.avatarUrl: /avatars/424905abb2973954b0850c592743b6fb.svg
paper_authors.fullname: Vikas Chandra
paper_authors.type: user
paper_authors.user: vchandra
paper_title: An Introduction to Vision-Language Modeling
paper_summary: Following the recent popularity of Large Language Models (LLMs), several
  attempts have been made to extend them to the visual domain. From having a
  visual assistant that could guide us through unfamiliar environments to
  generative models that produce images using only a high-level text description,
  the vision-language model (VLM) applications will significantly impact our
  relationship with technology. However, there are many challenges that need to
  be addressed to improve the reliability of those models. While language is
  discrete, vision evolves in a much higher dimensional space in which concepts
  cannot always be easily discretized. To better understand the mechanics behind
  mapping vision to language, we present this introduction to VLMs which we hope
  will help anyone who would like to enter the field. First, we introduce what
  VLMs are, how they work, and how to train them. Then, we present and discuss
  approaches to evaluate VLMs. Although this work primarily focuses on mapping
  images to language, we also discuss extending VLMs to videos.
paper_ai_summary: Introduction to vision-language models (VLMs) covering their applications, training, evaluation, and extension to videos, addressing challenges in mapping visual data to language.
paper_ai_keywords: Large Language Models (LLMs) vision-language models (VLMs) visual assistant generative models image to language vision discrete language high-dimensional space concept discretization video to language