title: PVChat: Personalized Video Chat with One-Shot Learning
summary: Video large language models (ViLLMs) excel in general video understanding,
  e.g., recognizing activities like talking and eating, but struggle with
  identity-aware comprehension, such as "Wilson is receiving chemotherapy" or
  "Tom is discussing with Sarah", limiting their applicability in smart
  healthcare and smart home environments. To address this limitation, we propose
  a one-shot learning framework PVChat, the first personalized ViLLM that enables
  subject-aware question answering (QA) from a single video for each subject. Our
  approach optimizes a Mixture-of-Heads (MoH) enhanced ViLLM on a synthetically
  augmented video-QA dataset, leveraging a progressive image-to-video learning
  strategy. Specifically, we introduce an automated augmentation pipeline that
  synthesizes identity-preserving positive samples and retrieves hard negatives
  from existing video corpora, generating a diverse training dataset with four QA
  types: existence, appearance, action, and location inquiries. To enhance
  subject-specific learning, we propose a ReLU Routing MoH attention mechanism,
  alongside two novel objectives: (1) Smooth Proximity Regularization for
  progressive learning through exponential distance scaling and (2) Head
  Activation Enhancement for balanced attention routing. Finally, we adopt a
  two-stage training strategy, transitioning from image pre-training to video
  fine-tuning, enabling a gradual learning process from static attributes to
  dynamic representations. We evaluate PVChat on diverse datasets covering
  medical scenarios, TV series, anime, and real-world footage, demonstrating its
  superiority in personalized feature understanding after learning from a single
  video, compared to state-of-the-art ViLLMs.
paper_authors._id: 67e0ba47753cfd5e438d3814
paper_authors.name: Yufei Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:10:50.772Z
paper_authors._id: 63be636387619d1458c2e8e0
paper_authors.avatarUrl: /avatars/83e14735760c5cadd5341ebcb4cf9556.svg
paper_authors.fullname: SHI YUFEI
paper_authors.type: user
paper_authors.user: Master-Shi
paper_authors._id: 67e0ba47753cfd5e438d3815
paper_authors.name: Weilong Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:54:43.276Z
paper_authors._id: 668192a3f35c3ff47a8438ee
paper_authors.avatarUrl: /avatars/6b293341f5dc51f574252c6f57cfd293.svg
paper_authors.fullname: Weilong Yan
paper_authors.type: user
paper_authors.user: DavidYan2001
paper_authors._id: 67e0ba47753cfd5e438d3816
paper_authors.name: Gang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:54:52.020Z
paper_authors._id: 676cbdc1dd95830fd9137137
paper_authors.avatarUrl: /avatars/71f36c630b475c08198d75ec377b6f3f.svg
paper_authors.fullname: xu
paper_authors.type: user
paper_authors.user: gangXu
paper_authors._id: 67e0ba47753cfd5e438d3817
paper_authors.name: Yumeng Li
paper_authors._id: 67e0ba47753cfd5e438d3818
paper_authors.name: Yuchen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:54:59.453Z
paper_authors._id: 646cdb564e59b82995df856b
paper_authors.avatarUrl: /avatars/4accb3f3d0289f8a5848e973ddcfcb92.svg
paper_authors.fullname: Jonathan Li
paper_authors.type: user
paper_authors.user: yumengli
paper_authors._id: 67e0ba47753cfd5e438d3819
paper_authors.name: Zhenxi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:55:27.186Z
paper_authors._id: 67198fff75da7b2981620711
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9pBj2JrSeYQHZg43jV6Fa.png
paper_authors.fullname: Zhenxi Li
paper_authors.type: user
paper_authors.user: wangxiangyu1
paper_authors._id: 67e0ba47753cfd5e438d381a
paper_authors.name: Fei Richard Yu
paper_authors._id: 67e0ba47753cfd5e438d381b
paper_authors.name: Ming Li
paper_authors._id: 67e0ba47753cfd5e438d381c
paper_authors.name: Si Yong Yeo
paper_title: PVChat: Personalized Video Chat with One-Shot Learning
paper_summary: Video large language models (ViLLMs) excel in general video understanding,
  e.g., recognizing activities like talking and eating, but struggle with
  identity-aware comprehension, such as "Wilson is receiving chemotherapy" or
  "Tom is discussing with Sarah", limiting their applicability in smart
  healthcare and smart home environments. To address this limitation, we propose
  a one-shot learning framework PVChat, the first personalized ViLLM that enables
  subject-aware question answering (QA) from a single video for each subject. Our
  approach optimizes a Mixture-of-Heads (MoH) enhanced ViLLM on a synthetically
  augmented video-QA dataset, leveraging a progressive image-to-video learning
  strategy. Specifically, we introduce an automated augmentation pipeline that
  synthesizes identity-preserving positive samples and retrieves hard negatives
  from existing video corpora, generating a diverse training dataset with four QA
  types: existence, appearance, action, and location inquiries. To enhance
  subject-specific learning, we propose a ReLU Routing MoH attention mechanism,
  alongside two novel objectives: (1) Smooth Proximity Regularization for
  progressive learning through exponential distance scaling and (2) Head
  Activation Enhancement for balanced attention routing. Finally, we adopt a
  two-stage training strategy, transitioning from image pre-training to video
  fine-tuning, enabling a gradual learning process from static attributes to
  dynamic representations. We evaluate PVChat on diverse datasets covering
  medical scenarios, TV series, anime, and real-world footage, demonstrating its
  superiority in personalized feature understanding after learning from a single
  video, compared to state-of-the-art ViLLMs.
paper_ai_summary: PVChat is a one-shot learning framework that enhances video large language models for personalized QA by introducing a ReLU Routing MoH attention mechanism and specific training objectives.
paper_ai_keywords: Mixture-of-Heads ReLU Routing MoH attention mechanism Smooth Proximity Regularization Head Activation Enhancement progressive image-to-video learning identity-preserving positive samples hard negatives existence appearance action location inquiries personalized ViLLM one-shot learning framework