title: PaliGemma 2: A Family of Versatile VLMs for Transfer
summary: PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM)
  based on the Gemma 2 family of language models. We combine the SigLIP-So400m
  vision encoder that was also used by PaliGemma with the whole range of Gemma 2
  models, from the 2B one all the way up to the 27B model. We train these models
  at three resolutions (224px, 448px, and 896px) in multiple stages to equip them
  with broad knowledge for transfer via fine-tuning. The resulting family of base
  models covering different model sizes and resolutions allows us to investigate
  factors impacting transfer performance (such as learning rate) and to analyze
  the interplay between the type of task, model size, and resolution. We further
  increase the number and breadth of transfer tasks beyond the scope of PaliGemma
  including different OCR-related tasks such as table structure recognition,
  molecular structure recognition, music score recognition, as well as long
  fine-grained captioning and radiography report generation, on which PaliGemma 2
  obtains state-of-the-art results.
paper_authors._id: 675159fd74b66f2e014ede09
paper_authors.name: Andreas Steiner
paper_authors._id: 675159fd74b66f2e014ede0a
paper_authors.name: André Susano Pinto
paper_authors._id: 675159fd74b66f2e014ede0b
paper_authors.name: Michael Tschannen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:40:57.297Z
paper_authors._id: 6489893e1ec8356ba5bb9777
paper_authors.avatarUrl: /avatars/54354c1e5774cadd1d83d42054e9d96b.svg
paper_authors.fullname: Michael Tschannen
paper_authors.type: user
paper_authors.user: mitsch
paper_authors._id: 675159fd74b66f2e014ede0c
paper_authors.name: Daniel Keysers
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:43:54.472Z
paper_authors._id: 662fc73ccee11621a1067db1
paper_authors.avatarUrl: /avatars/4fbfbc0621f631d8e95a2a642fa0cd27.svg
paper_authors.fullname: Daniel Keysers
paper_authors.type: user
paper_authors.user: dkeysers
paper_authors._id: 675159fd74b66f2e014ede0d
paper_authors.name: Xiao Wang
paper_authors._id: 675159fd74b66f2e014ede0e
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:40:46.933Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 675159fd74b66f2e014ede0f
paper_authors.name: Alexey Gritsenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:40:40.352Z
paper_authors._id: 62d8f9887b8dc0ba17271415
paper_authors.avatarUrl: /avatars/12ec78d34fd849bad44217b212f31e98.svg
paper_authors.fullname: Alexey Gritsenko
paper_authors.type: user
paper_authors.user: AlexeyG
paper_authors._id: 675159fd74b66f2e014ede10
paper_authors.name: Matthias Minderer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:40:35.142Z
paper_authors._id: 649ac6e57c36fc2dc6e6b0f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649ac6e57c36fc2dc6e6b0f4/Scd3LiYW5Y2qunDDHnPWw.jpeg
paper_authors.fullname: Matthias Minderer
paper_authors.type: user
paper_authors.user: mjlm
paper_authors._id: 675159fd74b66f2e014ede11
paper_authors.name: Anthony Sherbondy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:40:29.132Z
paper_authors._id: 65986c7fbf533e3c0dd00d83
paper_authors.avatarUrl: /avatars/d038ad9e50002ab4ef99e34d4ba369e7.svg
paper_authors.fullname: Anthony Sherbondy
paper_authors.type: user
paper_authors.user: asherbondy
paper_authors._id: 675159fd74b66f2e014ede12
paper_authors.name: Shangbang Long
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:40:23.703Z
paper_authors._id: 6525e3d136f98a576aa3d2a8
paper_authors.avatarUrl: /avatars/b5a12dd73ff05ae38c4d60d685db7c08.svg
paper_authors.fullname: ShangbangLong
paper_authors.type: user
paper_authors.user: Shangbang
paper_authors._id: 675159fd74b66f2e014ede13
paper_authors.name: Siyang Qin
paper_authors._id: 675159fd74b66f2e014ede14
paper_authors.name: Reeve Ingle
paper_authors._id: 675159fd74b66f2e014ede15
paper_authors.name: Emanuele Bugliarello
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:40:04.842Z
paper_authors._id: 626a9284783d5891f45beb53
paper_authors.avatarUrl: /avatars/8737662babc5d42cafba6087ab33e716.svg
paper_authors.fullname: Emanuele Bugliarello
paper_authors.type: user
paper_authors.user: e-bug
paper_authors._id: 675159fd74b66f2e014ede16
paper_authors.name: Sahar Kazemzadeh
paper_authors._id: 675159fd74b66f2e014ede17
paper_authors.name: Thomas Mesnard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:39:55.694Z
paper_authors._id: 6639adab8eb8c9c62ac3fb15
paper_authors.avatarUrl: /avatars/83133b04922143be1d8a1fb71fb2938f.svg
paper_authors.fullname: Thomas Mesnard
paper_authors.type: user
paper_authors.user: thomasmesnard
paper_authors._id: 675159fd74b66f2e014ede18
paper_authors.name: Ibrahim Alabdulmohsin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:39:49.864Z
paper_authors._id: 630545da20668afe24860235
paper_authors.avatarUrl: /avatars/5d82be2e7412bff1af15cc5eafa60b7d.svg
paper_authors.fullname: Ibrahim Alabdulmohsin
paper_authors.type: user
paper_authors.user: ibomohsin
paper_authors._id: 675159fd74b66f2e014ede19
paper_authors.name: Lucas Beyer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:39:44.061Z
paper_authors._id: 642d334ff65714b4585f2de4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
paper_authors.fullname: Lucas Beyer
paper_authors.type: user
paper_authors.user: giffmana
paper_authors._id: 675159fd74b66f2e014ede1a
paper_authors.name: Xiaohua Zhai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:39:18.732Z
paper_authors._id: 65dcd90082bddd501f68174b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/M2bc9PyKeFs1cCXjTfGGq.jpeg
paper_authors.fullname: Xiaohua Zhai
paper_authors.type: user
paper_authors.user: xiaohuazhai
paper_title: PaliGemma 2: A Family of Versatile VLMs for Transfer
paper_summary: PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM)
  based on the Gemma 2 family of language models. We combine the SigLIP-So400m
  vision encoder that was also used by PaliGemma with the whole range of Gemma 2
  models, from the 2B one all the way up to the 27B model. We train these models
  at three resolutions (224px, 448px, and 896px) in multiple stages to equip them
  with broad knowledge for transfer via fine-tuning. The resulting family of base
  models covering different model sizes and resolutions allows us to investigate
  factors impacting transfer performance (such as learning rate) and to analyze
  the interplay between the type of task, model size, and resolution. We further
  increase the number and breadth of transfer tasks beyond the scope of PaliGemma
  including different OCR-related tasks such as table structure recognition,
  molecular structure recognition, music score recognition, as well as long
  fine-grained captioning and radiography report generation, on which PaliGemma 2
  obtains state-of-the-art results.
paper_ai_keywords: SigLIP-So400m vision encoder language models fine-tuning resolution learning rate transfer performance OCR-related tasks table structure recognition molecular structure recognition music score recognition fine-grained captioning radiography report generation state-of-the-art results