title: Apollo: An Exploration of Video Understanding in Large Multimodal Models
summary: Despite the rapid integration of video perception capabilities into Large
  Multimodal Models (LMMs), the underlying mechanisms driving their video
  understanding remain poorly understood. Consequently, many design decisions in
  this domain are made without proper justification or analysis. The high
  computational cost of training and evaluating such models, coupled with limited
  open research, hinders the development of video-LMMs. To address this, we
  present a comprehensive study that helps uncover what effectively drives video
  understanding in LMMs.
    We begin by critically examining the primary contributors to the high
  computational requirements associated with video-LMM research and discover
  Scaling Consistency, wherein design and training decisions made on smaller
  models and datasets (up to a critical size) effectively transfer to larger
  models. Leveraging these insights, we explored many video-specific aspects of
  video-LMMs, including video sampling, architectures, data composition, training
  schedules, and more. For example, we demonstrated that fps sampling during
  training is vastly preferable to uniform frame sampling and which vision
  encoders are the best for video representation.
    Guided by these findings, we introduce Apollo, a state-of-the-art family of
  LMMs that achieve superior performance across different model sizes. Our models
  can perceive hour-long videos efficiently, with Apollo-3B outperforming most
  existing 7B models with an impressive 55.1 on LongVideoBench. Apollo-7B is
  state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on
  Video-MME.
paper_authors._id: 675f9d6d2eb87c3a1b120a5f
paper_authors.name: Orr Zohar
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-08T07:42:49.334Z
paper_authors._id: 648c9605565e3a44f3c9bb7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
paper_authors.fullname: Orr Zohar
paper_authors.type: user
paper_authors.user: orrzohar
paper_authors._id: 675f9d6d2eb87c3a1b120a60
paper_authors.name: Xiaohan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:23:12.868Z
paper_authors._id: 65703fab7f50602340d23704
paper_authors.avatarUrl: /avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
paper_authors.fullname: Xiaohan Wang
paper_authors.type: user
paper_authors.user: nicholswang
paper_authors._id: 675f9d6d2eb87c3a1b120a61
paper_authors.name: Yann Dubois
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T09:57:20.008Z
paper_authors._id: 641362754e5305c14f233f08
paper_authors.avatarUrl: /avatars/bce974ef60c507d22702cc7662033b28.svg
paper_authors.fullname: Yann Dubois
paper_authors.type: user
paper_authors.user: YannDubs
paper_authors._id: 675f9d6d2eb87c3a1b120a62
paper_authors.name: Nikhil Mehta
paper_authors._id: 675f9d6d2eb87c3a1b120a63
paper_authors.name: Tong Xiao
paper_authors._id: 675f9d6d2eb87c3a1b120a64
paper_authors.name: Philippe Hansen-Estruch
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:15.986Z
paper_authors._id: 644eb923a8f384abd038a4f7
paper_authors.avatarUrl: /avatars/3d1a29c6149a6ffc482842f1438ba55a.svg
paper_authors.fullname: Philippe I Hansen-Estruch
paper_authors.type: user
paper_authors.user: philippehansen
paper_authors._id: 675f9d6d2eb87c3a1b120a65
paper_authors.name: Licheng Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T09:58:30.800Z
paper_authors._id: 6372b1f9bd81fae2b3a712e8
paper_authors.avatarUrl: /avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
paper_authors.fullname: Licheng Yu
paper_authors.type: user
paper_authors.user: lichengyu
paper_authors._id: 675f9d6d2eb87c3a1b120a66
paper_authors.name: Xiaofang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T09:59:05.541Z
paper_authors._id: 6334c1b3762d2d0e9eb151ea
paper_authors.avatarUrl: /avatars/45b3eb63f0515ab417aacd79aa9f8a41.svg
paper_authors.fullname: Xiaofang Wang
paper_authors.type: user
paper_authors.user: minione
paper_authors._id: 675f9d6d2eb87c3a1b120a67
paper_authors.name: Felix Juefei-Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T16:55:18.689Z
paper_authors._id: 6417cf37dce1e4c0229f17b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
paper_authors.fullname: Felix Xu
paper_authors.type: user
paper_authors.user: katanaxu
paper_authors._id: 675f9d6d2eb87c3a1b120a68
paper_authors.name: Ning Zhang
paper_authors._id: 675f9d6d2eb87c3a1b120a69
paper_authors.name: Serena Yeung-Levy
paper_authors._id: 675f9d6d2eb87c3a1b120a6a
paper_authors.name: Xide Xia
paper_title: Apollo: An Exploration of Video Understanding in Large Multimodal Models
paper_summary: Despite the rapid integration of video perception capabilities into Large
  Multimodal Models (LMMs), the underlying mechanisms driving their video
  understanding remain poorly understood. Consequently, many design decisions in
  this domain are made without proper justification or analysis. The high
  computational cost of training and evaluating such models, coupled with limited
  open research, hinders the development of video-LMMs. To address this, we
  present a comprehensive study that helps uncover what effectively drives video
  understanding in LMMs.
    We begin by critically examining the primary contributors to the high
  computational requirements associated with video-LMM research and discover
  Scaling Consistency, wherein design and training decisions made on smaller
  models and datasets (up to a critical size) effectively transfer to larger
  models. Leveraging these insights, we explored many video-specific aspects of
  video-LMMs, including video sampling, architectures, data composition, training
  schedules, and more. For example, we demonstrated that fps sampling during
  training is vastly preferable to uniform frame sampling and which vision
  encoders are the best for video representation.
    Guided by these findings, we introduce Apollo, a state-of-the-art family of
  LMMs that achieve superior performance across different model sizes. Our models
  can perceive hour-long videos efficiently, with Apollo-3B outperforming most
  existing 7B models with an impressive 55.1 on LongVideoBench. Apollo-7B is
  state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on
  Video-MME.
paper_ai_summary: A comprehensive study reveals effective design principles for video understanding in Large Multimodal Models, introducing Apollo, which achieves superior performance across different model sizes.
paper_ai_keywords: LMMs video understanding computational requirements Scaling Consistency video sampling architectures data composition training schedules fps sampling vision encoders Apollo LongVideoBench MLVU Video-MME