PVChat: Personalized Video Chat with One-Shot Learning
Video large language models (ViLLMs) excel in general video understanding,
e.g., recognizing activities like talking and eating, but struggle with
identity-aware comprehension, such as "Wilson is receiving chemotherapy" or
"Tom is discussing with Sarah", limiting their applicability in smart
healthcare and smart home environments. To address this limitation, we propose
a one-shot learning framework PVChat, the first personalized ViLLM that enables
subject-aware question answering (QA) from a single video for each subject. Our
approach optimizes a Mixture-of-Heads (MoH) enhanced ViLLM on a synthetically
augmented video-QA dataset, leveraging a progressive image-to-video learning
strategy. Specifically, we introduce an automated augmentation pipeline that
synthesizes identity-preserving positive samples and retrieves hard negatives
from existing video corpora, generating a diverse training dataset with four QA
types: existence, appearance, action, and location inquiries. To enhance
subject-specific learning, we propose a ReLU Routing MoH attention mechanism,
alongside two novel objectives: (1) Smooth Proximity Regularization for
progressive learning through exponential distance scaling and (2) Head
Activation Enhancement for balanced attention routing. Finally, we adopt a
two-stage training strategy, transitioning from image pre-training to video
fine-tuning, enabling a gradual learning process from static attributes to
dynamic representations. We evaluate PVChat on diverse datasets covering
medical scenarios, TV series, anime, and real-world footage, demonstrating its
superiority in personalized feature understanding after learning from a single
video, compared to state-of-the-art ViLLMs.
67e0ba47753cfd5e438d3814
Yufei Shi
claimed_verified
2025-03-24T08:10:50.772Z
63be636387619d1458c2e8e0
/avatars/83e14735760c5cadd5341ebcb4cf9556.svg
SHI YUFEI
user
Master-Shi
67e0ba47753cfd5e438d3815
Weilong Yan
admin_assigned
2025-03-24T12:54:43.276Z
668192a3f35c3ff47a8438ee
/avatars/6b293341f5dc51f574252c6f57cfd293.svg
Weilong Yan
user
DavidYan2001
67e0ba47753cfd5e438d3816
Gang Xu
admin_assigned
2025-03-24T12:54:52.020Z
676cbdc1dd95830fd9137137
/avatars/71f36c630b475c08198d75ec377b6f3f.svg
xu
user
gangXu
67e0ba47753cfd5e438d3817
Yumeng Li
67e0ba47753cfd5e438d3818
Yuchen Li
admin_assigned
2025-03-24T12:54:59.453Z
646cdb564e59b82995df856b
/avatars/4accb3f3d0289f8a5848e973ddcfcb92.svg
Jonathan Li
user
yumengli
67e0ba47753cfd5e438d3819
Zhenxi Li
admin_assigned
2025-03-24T12:55:27.186Z
67198fff75da7b2981620711
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9pBj2JrSeYQHZg43jV6Fa.png
Zhenxi Li
user
wangxiangyu1
67e0ba47753cfd5e438d381a
Fei Richard Yu
67e0ba47753cfd5e438d381b
Ming Li
67e0ba47753cfd5e438d381c
Si Yong Yeo
PVChat: Personalized Video Chat with One-Shot Learning
Video large language models (ViLLMs) excel in general video understanding,
e.g., recognizing activities like talking and eating, but struggle with
identity-aware comprehension, such as "Wilson is receiving chemotherapy" or
"Tom is discussing with Sarah", limiting their applicability in smart
healthcare and smart home environments. To address this limitation, we propose
a one-shot learning framework PVChat, the first personalized ViLLM that enables
subject-aware question answering (QA) from a single video for each subject. Our
approach optimizes a Mixture-of-Heads (MoH) enhanced ViLLM on a synthetically
augmented video-QA dataset, leveraging a progressive image-to-video learning
strategy. Specifically, we introduce an automated augmentation pipeline that
synthesizes identity-preserving positive samples and retrieves hard negatives
from existing video corpora, generating a diverse training dataset with four QA
types: existence, appearance, action, and location inquiries. To enhance
subject-specific learning, we propose a ReLU Routing MoH attention mechanism,
alongside two novel objectives: (1) Smooth Proximity Regularization for
progressive learning through exponential distance scaling and (2) Head
Activation Enhancement for balanced attention routing. Finally, we adopt a
two-stage training strategy, transitioning from image pre-training to video
fine-tuning, enabling a gradual learning process from static attributes to
dynamic representations. We evaluate PVChat on diverse datasets covering
medical scenarios, TV series, anime, and real-world footage, demonstrating its
superiority in personalized feature understanding after learning from a single
video, compared to state-of-the-art ViLLMs.
PVChat is a one-shot learning framework that enhances video large language models for personalized QA by introducing a ReLU Routing MoH attention mechanism and specific training objectives.
Mixture-of-Heads ReLU Routing MoH attention mechanism Smooth Proximity Regularization Head Activation Enhancement progressive image-to-video learning identity-preserving positive samples hard negatives existence appearance action location inquiries personalized ViLLM one-shot learning framework