title: PiTe: Pixel-Temporal Alignment for Large Video-Language Model
summary: Fueled by the Large Language Models (LLMs) wave, Large Visual-Language Models
  (LVLMs) have emerged as a pivotal advancement, bridging the gap between image
  and text. However, video making it challenging for LVLMs to perform adequately
  due to the complexity of the relationship between language and spatial-temporal
  data structure. Recent Large Video-Language Models (LVidLMs) align feature of
  static visual data like image into latent space of language feature, by general
  multi-modal tasks to leverage abilities of LLMs sufficiently. In this paper, we
  explore fine-grained alignment approach via object trajectory for different
  modalities across both spatial and temporal dimensions simultaneously. Thus, we
  propose a novel LVidLM by trajectory-guided Pixel-Temporal Alignment, dubbed
  PiTe, that exhibits promising applicable model property. To achieve
  fine-grained video-language alignment, we curate a multi-modal pre-training
  dataset PiTe-143k, the dataset provision of moving trajectories in pixel level
  for all individual objects, that appear and mention in the video and caption
  both, by our automatic annotation pipeline. Meanwhile, PiTe demonstrates
  astounding capabilities on myriad video-related multi-modal tasks through beat
  the state-of-the-art methods by a large margin.
paper_authors._id: 66e40348038300b07a7dda0f
paper_authors.name: Yang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T09:10:53.960Z
paper_authors._id: 6586817e509bcae23f3dfc60
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pwEStoey1XJYDi3F0Z930.png
paper_authors.fullname: Yang Liu
paper_authors.type: user
paper_authors.user: yliu-cs
paper_authors._id: 66e40348038300b07a7dda10
paper_authors.name: Pengxiang Ding
paper_authors._id: 66e40348038300b07a7dda11
paper_authors.name: Siteng Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-13T12:09:08.813Z
paper_authors._id: 65fd82762bf2cd20ddaa193f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yBYbWp_mT7UusYdkqtAvw.png
paper_authors.fullname: Siteng Huang
paper_authors.type: user
paper_authors.user: huangsiteng
paper_authors._id: 66e40348038300b07a7dda12
paper_authors.name: Min Zhang
paper_authors._id: 66e40348038300b07a7dda13
paper_authors.name: Han Zhao
paper_authors._id: 66e40348038300b07a7dda14
paper_authors.name: Donglin Wang
paper_title: PiTe: Pixel-Temporal Alignment for Large Video-Language Model
paper_summary: Fueled by the Large Language Models (LLMs) wave, Large Visual-Language Models
  (LVLMs) have emerged as a pivotal advancement, bridging the gap between image
  and text. However, video making it challenging for LVLMs to perform adequately
  due to the complexity of the relationship between language and spatial-temporal
  data structure. Recent Large Video-Language Models (LVidLMs) align feature of
  static visual data like image into latent space of language feature, by general
  multi-modal tasks to leverage abilities of LLMs sufficiently. In this paper, we
  explore fine-grained alignment approach via object trajectory for different
  modalities across both spatial and temporal dimensions simultaneously. Thus, we
  propose a novel LVidLM by trajectory-guided Pixel-Temporal Alignment, dubbed
  PiTe, that exhibits promising applicable model property. To achieve
  fine-grained video-language alignment, we curate a multi-modal pre-training
  dataset PiTe-143k, the dataset provision of moving trajectories in pixel level
  for all individual objects, that appear and mention in the video and caption
  both, by our automatic annotation pipeline. Meanwhile, PiTe demonstrates
  astounding capabilities on myriad video-related multi-modal tasks through beat
  the state-of-the-art methods by a large margin.
paper_ai_summary: A new video-language model, PiTe, uses trajectory-guided pixel-temporal alignment to achieve superior performance across various multimodal video tasks by leveraging a large pre-training dataset with precise object trajectories.
paper_ai_keywords: Large Language Models (LLMs) Large Visual-Language Models (LVLMs) video spatial-temporal data Large Video-Language Models (LVidLMs) latent space multi-modal tasks fine-grained alignment object trajectory trajectory-guided Pixel-Temporal Alignment PiTe pre-training dataset PiTe-143k automatic annotation pipeline