title: Tarsier2: Advancing Large Vision-Language Models from Detailed Video
    Description to Comprehensive Video Understanding
summary: We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM)
  designed for generating detailed and accurate video descriptions, while also
  exhibiting superior general video understanding capabilities. Tarsier2 achieves
  significant advancements through three key upgrades: (1) Scaling pre-training
  data from 11M to 40M video-text pairs, enriching both volume and diversity; (2)
  Performing fine-grained temporal alignment during supervised fine-tuning; (3)
  Using model-based sampling to automatically construct preference data and
  applying DPO training for optimization. Extensive experiments show that
  Tarsier2-7B consistently outperforms leading proprietary models, including
  GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K
  benchmark, Tarsier2-7B improves F1 by 2.8\% over GPT-4o and 5.8\% over
  Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6\%
  performance advantage over GPT-4o and +24.9\% over Gemini-1.5-Pro. Tarsier2-7B
  also sets new state-of-the-art results across 15 public benchmarks, spanning
  tasks such as video question-answering, video grounding, hallucination test,
  and embodied question-answering, demonstrating its versatility as a robust
  generalist vision-language model.
paper_authors._id: 67873cdd1f65db189f9f64d7
paper_authors.name: Liping Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T17:07:12.581Z
paper_authors._id: 6638cd255b81e56d337a4a98
paper_authors.avatarUrl: /avatars/b00805982755e4838b5ce0b23e3357c2.svg
paper_authors.fullname: Liping Yuan
paper_authors.type: user
paper_authors.user: yuanlp
paper_authors._id: 67873cdd1f65db189f9f64d8
paper_authors.name: Jiawei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T21:21:17.147Z
paper_authors._id: 6788122301d5ba1d3eff23ba
paper_authors.avatarUrl: /avatars/203dc8e1d542be55ea16eafcd7f396ff.svg
paper_authors.fullname: Jiawei Wang
paper_authors.type: user
paper_authors.user: 0nejiawei
paper_authors._id: 67873cdd1f65db189f9f64d9
paper_authors.name: Haomiao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T17:07:46.243Z
paper_authors._id: 668685161589885791b76c25
paper_authors.avatarUrl: /avatars/ffb9339114b75205ab5840b70f59f4e5.svg
paper_authors.fullname: sunhaomiao
paper_authors.type: user
paper_authors.user: sunhm15
paper_authors._id: 67873cdd1f65db189f9f64da
paper_authors.name: Yuchen Zhang
paper_authors._id: 67873cdd1f65db189f9f64db
paper_authors.name: Yuan Lin
paper_title: Tarsier2: Advancing Large Vision-Language Models from Detailed Video
    Description to Comprehensive Video Understanding
paper_summary: We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM)
  designed for generating detailed and accurate video descriptions, while also
  exhibiting superior general video understanding capabilities. Tarsier2 achieves
  significant advancements through three key upgrades: (1) Scaling pre-training
  data from 11M to 40M video-text pairs, enriching both volume and diversity; (2)
  Performing fine-grained temporal alignment during supervised fine-tuning; (3)
  Using model-based sampling to automatically construct preference data and
  applying DPO training for optimization. Extensive experiments show that
  Tarsier2-7B consistently outperforms leading proprietary models, including
  GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K
  benchmark, Tarsier2-7B improves F1 by 2.8\% over GPT-4o and 5.8\% over
  Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6\%
  performance advantage over GPT-4o and +24.9\% over Gemini-1.5-Pro. Tarsier2-7B
  also sets new state-of-the-art results across 15 public benchmarks, spanning
  tasks such as video question-answering, video grounding, hallucination test,
  and embodied question-answering, demonstrating its versatility as a robust
  generalist vision-language model.
paper_ai_summary: Tarsier2, a large vision-language model, achieves state-of-the-art performance in video description and understanding by scaling pre-training data, fine-grained temporal alignment, and using DPO training.
paper_ai_keywords: large vision-language model video description fine-grained temporal alignment model-based sampling DPO training video question-answering video grounding hallucination test embodied question-answering