title: Molmo and PixMo: Open Weights and Open Data for State-of-the-Art
    Multimodal Models
summary: Today's most advanced multimodal models remain proprietary. The strongest
  open-weight models rely heavily on synthetic data from proprietary VLMs to
  achieve good performance, effectively distilling these closed models into open
  ones. As a result, the community is still missing foundational knowledge about
  how to build performant VLMs from scratch. We present Molmo, a new family of
  VLMs that are state-of-the-art in their class of openness. Our key innovation
  is a novel, highly detailed image caption dataset collected entirely from human
  annotators using speech-based descriptions. To enable a wide array of user
  interactions, we also introduce a diverse dataset mixture for fine-tuning that
  includes in-the-wild Q&A and innovative 2D pointing data. The success of our
  approach relies on careful choices for the model architecture details, a
  well-tuned training pipeline, and, most critically, the quality of our newly
  collected datasets, all of which will be released. The best-in-class 72B model
  within the Molmo family not only outperforms others in the class of open weight
  and data models but also compares favorably against proprietary systems like
  GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human
  evaluation.
    We will be releasing all of our model weights, captioning and fine-tuning
  data, and source code in the near future. Select model weights, inference code,
  and demo are available at https://molmo.allenai.org.
paper_authors._id: 66f4c5c9b26f10a31416f516
paper_authors.name: Matt Deitke
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:23:04.074Z
paper_authors._id: 61c388aa727d1257bf3cf58b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670871898141-61c388aa727d1257bf3cf58b.jpeg
paper_authors.fullname: Matt Deitke
paper_authors.type: user
paper_authors.user: mattdeitke
paper_authors._id: 66f4c5c9b26f10a31416f517
paper_authors.name: Christopher Clark
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T20:11:33.431Z
paper_authors._id: 65c92082f3764ed670119351
paper_authors.avatarUrl: /avatars/85ebdd196215175d07397ad2c15a2a9b.svg
paper_authors.fullname: Christopher Clark
paper_authors.type: user
paper_authors.user: chrisc36
paper_authors._id: 66f4c5c9b26f10a31416f518
paper_authors.name: Sangho Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:08.524Z
paper_authors._id: 664fc02d6f16bbd9a168654d
paper_authors.avatarUrl: /avatars/9f726ede9973706bc198845464238452.svg
paper_authors.fullname: Sangho Lee
paper_authors.type: user
paper_authors.user: sanghol
paper_authors._id: 66f4c5c9b26f10a31416f519
paper_authors.name: Rohun Tripathi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:19:34.767Z
paper_authors._id: 646e5aa2cb6ea6e6b6df2a5f
paper_authors.avatarUrl: /avatars/65c63a138d793a17fc317fccc2be53b2.svg
paper_authors.fullname: Rohun Tripathi
paper_authors.type: user
paper_authors.user: roTripathi
paper_authors._id: 66f4c5c9b26f10a31416f51a
paper_authors.name: Yue Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T12:49:36.119Z
paper_authors._id: 62f6c68904e5e02f82b04690
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
paper_authors.fullname: Yue Yang
paper_authors.type: user
paper_authors.user: yyupenn
paper_authors._id: 66f4c5c9b26f10a31416f51b
paper_authors.name: Jae Sung Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:20:14.540Z
paper_authors._id: 6345a11743f4f2d2ed1057ca
paper_authors.avatarUrl: /avatars/cba8b194b2517da76021d8cc1814a542.svg
paper_authors.fullname: Jae Sung Park
paper_authors.type: user
paper_authors.user: jamepark3922
paper_authors._id: 66f4c5c9b26f10a31416f51c
paper_authors.name: Mohammadreza Salehi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:20:26.387Z
paper_authors._id: 66990ec3aa500cd99d640a19
paper_authors.avatarUrl: /avatars/3520cab2b946158b2cdf6ed84f9516e0.svg
paper_authors.fullname: Mohammadreza Salehi
paper_authors.type: user
paper_authors.user: SMSD75
paper_authors._id: 66f4c5c9b26f10a31416f51d
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:20:32.338Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 66f4c5c9b26f10a31416f51e
paper_authors.name: Kyle Lo
paper_authors._id: 66f4c5c9b26f10a31416f51f
paper_authors.name: Luca Soldaini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:20:44.007Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 66f4c5c9b26f10a31416f520
paper_authors.name: Jiasen Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:20:50.472Z
paper_authors._id: 62b6b0397523238923221df9
paper_authors.avatarUrl: /avatars/77068771dd51df7519516cd502a88789.svg
paper_authors.fullname: Jiasenlu
paper_authors.type: user
paper_authors.user: Jiasenlu
paper_authors._id: 66f4c5c9b26f10a31416f521
paper_authors.name: Taira Anderson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:20:56.203Z
paper_authors._id: 65de20ad4e73a7dea7fb4f08
paper_authors.avatarUrl: /avatars/f3b0ad6cc9417e8ea3f0607fa62824d1.svg
paper_authors.fullname: Taira Anderson
paper_authors.type: user
paper_authors.user: tairaa
paper_authors._id: 66f4c5c9b26f10a31416f522
paper_authors.name: Erin Bransom
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:21:03.691Z
paper_authors._id: 65bd91cfb54ab5b37dd1e309
paper_authors.avatarUrl: /avatars/33a26756df8275b94e8e4f1f8a6e8cbc.svg
paper_authors.fullname: Erin Bransom
paper_authors.type: user
paper_authors.user: erinbransom
paper_authors._id: 66f4c5c9b26f10a31416f523
paper_authors.name: Kiana Ehsani
paper_authors._id: 66f4c5c9b26f10a31416f524
paper_authors.name: Huong Ngo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:21:27.740Z
paper_authors._id: 66e89c8525e09fb294c6ad3b
paper_authors.avatarUrl: /avatars/7044e2696487f20b2fe50967d805c43b.svg
paper_authors.fullname: Huong Ngo
paper_authors.type: user
paper_authors.user: huongn
paper_authors._id: 66f4c5c9b26f10a31416f525
paper_authors.name: YenSung Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:21:37.323Z
paper_authors._id: 6697093a37d2483826562c24
paper_authors.avatarUrl: /avatars/0e526b4be6db07e2485f7ef862080339.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Yensung
paper_authors._id: 66f4c5c9b26f10a31416f526
paper_authors.name: Ajay Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T07:38:27.736Z
paper_authors._id: 61c40eeb727d1257bf3cf5ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c40eeb727d1257bf3cf5ba/hVNbcFjsvwWqWarcGTOdI.jpeg
paper_authors.fullname: Ajay Patel
paper_authors.type: user
paper_authors.user: AjayP13
paper_authors._id: 66f4c5c9b26f10a31416f527
paper_authors.name: Mark Yatskar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:22:06.633Z
paper_authors._id: 631f42e1b6628770f6efd87a
paper_authors.avatarUrl: /avatars/0fa93b3513ebca737cce26dfa5611cf1.svg
paper_authors.fullname: Mark Yatskar
paper_authors.type: user
paper_authors.user: myatskar
paper_authors._id: 66f4c5c9b26f10a31416f528
paper_authors.name: Chris Callison-Burch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:22:13.265Z
paper_authors._id: 6303ce25fc783bfc744216af
paper_authors.avatarUrl: /avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg
paper_authors.fullname: Chris Callison-Burch
paper_authors.type: user
paper_authors.user: CCB
paper_authors._id: 66f4c5c9b26f10a31416f529
paper_authors.name: Andrew Head
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:22:19.015Z
paper_authors._id: 6360289588e41d249ecd3e26
paper_authors.avatarUrl: /avatars/0e84dbd72f07e99967a8d25cda938efe.svg
paper_authors.fullname: Andrew Head
paper_authors.type: user
paper_authors.user: ChittyChins
paper_authors._id: 66f4c5c9b26f10a31416f52a
paper_authors.name: Rose Hendrix
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:22:24.427Z
paper_authors._id: 627598f47128d87e9f92f487
paper_authors.avatarUrl: /avatars/1f9ae7236b25f573f62e298436c7422a.svg
paper_authors.fullname: Rose Hendrix
paper_authors.type: user
paper_authors.user: roseh
paper_authors._id: 66f4c5c9b26f10a31416f52b
paper_authors.name: Favyen Bastani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:22:29.774Z
paper_authors._id: 64dbe43c70446182be8727a8
paper_authors.avatarUrl: /avatars/4390ef620010911901ebb69d944e6b98.svg
paper_authors.fullname: Favyen Bastani
paper_authors.type: user
paper_authors.user: favyen
paper_authors._id: 66f4c5c9b26f10a31416f52c
paper_authors.name: Eli VanderBilt
paper_authors._id: 66f4c5c9b26f10a31416f52d
paper_authors.name: Nathan Lambert
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-06T08:01:02.562Z
paper_authors._id: 628e5f90a9a3c754c1f7c88f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
paper_authors.fullname: Nathan Lambert
paper_authors.type: user
paper_authors.user: natolambert
paper_authors._id: 66f4c5c9b26f10a31416f52e
paper_authors.name: Yvonne Chou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:28:48.664Z
paper_authors._id: 647a38bad547ef74714ce526
paper_authors.avatarUrl: /avatars/070c3718078c35046b2fc09ed16ae8f2.svg
paper_authors.fullname: Yvonne Chou
paper_authors.type: user
paper_authors.user: ychou-ai2
paper_authors._id: 66f4c5c9b26f10a31416f52f
paper_authors.name: Arnavi Chheda
paper_authors._id: 66f4c5c9b26f10a31416f530
paper_authors.name: Jenna Sparks
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:29:31.683Z
paper_authors._id: 65cd45626f58553a7447b342
paper_authors.avatarUrl: /avatars/7dacfd296b99822f13a61a09c06aca1c.svg
paper_authors.fullname: Jenna Sparks
paper_authors.type: user
paper_authors.user: jennasparks
paper_authors._id: 66f4c5c9b26f10a31416f531
paper_authors.name: Sam Skjonsberg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:29:38.399Z
paper_authors._id: 603572ed7d20da1ddcb2f59b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/603572ed7d20da1ddcb2f59b/f4suP0L1vzQsJ0OV7WDrS.png
paper_authors.fullname: Sam Skjonsberg
paper_authors.type: user
paper_authors.user: codeviking
paper_authors._id: 66f4c5c9b26f10a31416f532
paper_authors.name: Michael Schmitz
paper_authors._id: 66f4c5c9b26f10a31416f533
paper_authors.name: Aaron Sarnat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:26:25.270Z
paper_authors._id: 65b1520bf7638a13a641a620
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b1520bf7638a13a641a620/KTauZL0kXlmYnbkI2lFBG.png
paper_authors.fullname: Aaron Sarnat
paper_authors.type: user
paper_authors.user: aaronsarnat
paper_authors._id: 66f4c5c9b26f10a31416f534
paper_authors.name: Byron Bischoff
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:26:19.505Z
paper_authors._id: 67f85d757f56304dc72b4961
paper_authors.avatarUrl: /avatars/6ae806d25632b250220145feae200058.svg
paper_authors.fullname: Byron Bischoff
paper_authors.type: user
paper_authors.user: byronbischoff
paper_authors._id: 66f4c5c9b26f10a31416f535
paper_authors.name: Pete Walsh
paper_authors._id: 66f4c5c9b26f10a31416f536
paper_authors.name: Chris Newell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:25:39.106Z
paper_authors._id: 669fe6013db54f36c2b1198b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KY10scX-CtLoDF9oITKeY.jpeg
paper_authors.fullname: Chris Newell
paper_authors.type: user
paper_authors.user: newellce
paper_authors._id: 66f4c5c9b26f10a31416f537
paper_authors.name: Piper Wolters
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:25:25.912Z
paper_authors._id: 6509e8202e4bbde418147207
paper_authors.avatarUrl: /avatars/b8926cb9f5f1a8cff36432ae18ee3c32.svg
paper_authors.fullname: Piper Wolters
paper_authors.type: user
paper_authors.user: piperw
paper_authors._id: 66f4c5c9b26f10a31416f538
paper_authors.name: Tanmay Gupta
paper_authors._id: 66f4c5c9b26f10a31416f539
paper_authors.name: Kuo-Hao Zeng
paper_authors._id: 66f4c5c9b26f10a31416f53a
paper_authors.name: Jon Borchardt
paper_authors._id: 66f4c5c9b26f10a31416f53b
paper_authors.name: Dirk Groeneveld
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:24:41.590Z
paper_authors._id: 60369745413a78f892e7339c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
paper_authors.fullname: Dirk Groeneveld
paper_authors.type: user
paper_authors.user: dirkgr
paper_authors._id: 66f4c5c9b26f10a31416f53c
paper_authors.name: Jen Dumas
paper_authors._id: 66f4c5c9b26f10a31416f53d
paper_authors.name: Crystal Nam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:24:19.946Z
paper_authors._id: 64a7264f44387a55046f92e5
paper_authors.avatarUrl: /avatars/7f6e076a2a535957bc9c7ac4aeae716d.svg
paper_authors.fullname: Crystal Nam
paper_authors.type: user
paper_authors.user: CrystalN
paper_authors._id: 66f4c5c9b26f10a31416f53e
paper_authors.name: Sophie Lebrecht
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:24:12.561Z
paper_authors._id: 65b301f04c9e50e74a893954
paper_authors.avatarUrl: /avatars/f52366959f9e7613576603c0272ff2c5.svg
paper_authors.fullname: Sophie Lebrecht
paper_authors.type: user
paper_authors.user: Lebrechts
paper_authors._id: 66f4c5c9b26f10a31416f53f
paper_authors.name: Caitlin Wittlif
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:24:05.433Z
paper_authors._id: 652db071b62cf1f8463221e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/-GDOf0IQdomYEEjFD1cMz.png
paper_authors.fullname: Caitlin Wittlif
paper_authors.type: user
paper_authors.user: caitwallenai
paper_authors._id: 66f4c5c9b26f10a31416f540
paper_authors.name: Carissa Schoenick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:23:58.908Z
paper_authors._id: 6024546dc1f3c79f98e4b384
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1612993792778-6024546dc1f3c79f98e4b384.jpeg
paper_authors.fullname: Carissa Schoenick
paper_authors.type: user
paper_authors.user: CarissaS
paper_authors._id: 66f4c5c9b26f10a31416f541
paper_authors.name: Oscar Michel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:23:37.182Z
paper_authors._id: 633e387b309a99325093c6ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665022070673-noauth.jpeg
paper_authors.fullname: Oscar  Michel
paper_authors.type: user
paper_authors.user: breadroll
paper_authors._id: 66f4c5c9b26f10a31416f542
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:23:29.387Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_authors._id: 66f4c5c9b26f10a31416f543
paper_authors.name: Luca Weihs
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:23:19.856Z
paper_authors._id: 620acbdb3c0931626a7c9297
paper_authors.avatarUrl: /avatars/f63b1d225ed81e223d3e8876a5c708c4.svg
paper_authors.fullname: Luca Weihs
paper_authors.type: user
paper_authors.user: lucaweihs
paper_authors._id: 66f4c5c9b26f10a31416f544
paper_authors.name: Noah A. Smith
paper_authors._id: 66f4c5c9b26f10a31416f545
paper_authors.name: Hannaneh Hajishirzi
paper_authors._id: 66f4c5c9b26f10a31416f546
paper_authors.name: Ross Girshick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:23:04.745Z
paper_authors._id: 645a98a777d040cfd6500e14
paper_authors.avatarUrl: /avatars/9695beb63f88952064936c7345d09ee2.svg
paper_authors.fullname: Ross Girshick
paper_authors.type: user
paper_authors.user: inkynumbers
paper_authors._id: 66f4c5c9b26f10a31416f547
paper_authors.name: Ali Farhadi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T09:22:58.044Z
paper_authors._id: 6660d4c1818c5c5ca0f31266
paper_authors.avatarUrl: /avatars/1d2972894cb3b9df1900fdb162d9c364.svg
paper_authors.fullname: alifarhadi 
paper_authors.type: user
paper_authors.user: alifarhadi051
paper_authors._id: 66f4c5c9b26f10a31416f548
paper_authors.name: Aniruddha Kembhavi
paper_title: Molmo and PixMo: Open Weights and Open Data for State-of-the-Art
    Multimodal Models
paper_summary: Today's most advanced multimodal models remain proprietary. The strongest
  open-weight models rely heavily on synthetic data from proprietary VLMs to
  achieve good performance, effectively distilling these closed models into open
  ones. As a result, the community is still missing foundational knowledge about
  how to build performant VLMs from scratch. We present Molmo, a new family of
  VLMs that are state-of-the-art in their class of openness. Our key innovation
  is a novel, highly detailed image caption dataset collected entirely from human
  annotators using speech-based descriptions. To enable a wide array of user
  interactions, we also introduce a diverse dataset mixture for fine-tuning that
  includes in-the-wild Q&A and innovative 2D pointing data. The success of our
  approach relies on careful choices for the model architecture details, a
  well-tuned training pipeline, and, most critically, the quality of our newly
  collected datasets, all of which will be released. The best-in-class 72B model
  within the Molmo family not only outperforms others in the class of open weight
  and data models but also compares favorably against proprietary systems like
  GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human
  evaluation.
    We will be releasing all of our model weights, captioning and fine-tuning
  data, and source code in the near future. Select model weights, inference code,
  and demo are available at https://molmo.allenai.org.
paper_ai_summary: Molmo, a new family of open-weight VLMs, excels through a high-quality human-annotated image caption dataset and a diverse fine-tuning dataset, outperforming both proprietary and open models in benchmark and human evaluations.
paper_ai_keywords: multimodal models open-weight models VLMs image caption dataset human annotators speech-based descriptions 2D pointing data model architecture training pipeline GPT-4o Claude 3.5 Gemini 1.5