HAIC: Improving Human Action Understanding and Generation with Better
  Captions for Multi-modal Large Language Models
Recent Multi-modal Large Language Models (MLLMs) have made great progress in
video understanding. However, their performance on videos involving human
actions is still limited by the lack of high-quality data. To address this, we
introduce a two-stage data annotation pipeline. First, we design strategies to
accumulate videos featuring clear human actions from the Internet. Second,
videos are annotated in a standardized caption format that uses human
attributes to distinguish individuals and chronologically details their actions
and interactions. Through this pipeline, we curate two datasets, namely
HAICTrain and HAICBench. HAICTrain comprises 126K video-caption pairs
generated by Gemini-Pro and verified for training purposes. Meanwhile,
HAICBench includes 500 manually annotated video-caption pairs and
1,400 QA pairs, for a comprehensive evaluation of human action understanding.
Experimental results demonstrate that training with HAICTrain not only
significantly enhances human understanding abilities across 4 benchmarks, but
can also improve text-to-video generation results. Both the HAICTrain and
HAICBench are released at https://huggingface.co/datasets/KuaishouHAIC/HAIC.
67c51c198d02783fa3a6249d
Xiao Wang
67c51c198d02783fa3a6249e
Jingyun Hua
claimed_verified
2025-07-03T07:43:34.597Z
61540338e5b9ae6774201e58
https://cdn-avatars.huggingface.co/v1/production/uploads/61540338e5b9ae6774201e58/p_minqil1sdiqg5wEVxT5.jpeg
jingyun
user
hjy
67c51c198d02783fa3a6249f
Weihong Lin
admin_assigned
2025-03-03T09:42:30.547Z
675a69699e086bd6250a36ef
/avatars/95c72e3975d1a37f8655a2fe629746ec.svg
Weihong Lin
user
lwher1996
67c51c198d02783fa3a624a0
Yuanxing Zhang
67c51c198d02783fa3a624a1
Fuzheng Zhang
67c51c198d02783fa3a624a2
Jianlong Wu
67c51c198d02783fa3a624a3
Di Zhang
67c51c198d02783fa3a624a4
Liqiang Nie
HAIC: Improving Human Action Understanding and Generation with Better
  Captions for Multi-modal Large Language Models
Recent Multi-modal Large Language Models (MLLMs) have made great progress in
video understanding. However, their performance on videos involving human
actions is still limited by the lack of high-quality data. To address this, we
introduce a two-stage data annotation pipeline. First, we design strategies to
accumulate videos featuring clear human actions from the Internet. Second,
videos are annotated in a standardized caption format that uses human
attributes to distinguish individuals and chronologically details their actions
and interactions. Through this pipeline, we curate two datasets, namely
HAICTrain and HAICBench. HAICTrain comprises 126K video-caption pairs
generated by Gemini-Pro and verified for training purposes. Meanwhile,
HAICBench includes 500 manually annotated video-caption pairs and
1,400 QA pairs, for a comprehensive evaluation of human action understanding.
Experimental results demonstrate that training with HAICTrain not only
significantly enhances human understanding abilities across 4 benchmarks, but
can also improve text-to-video generation results. Both the HAICTrain and
HAICBench are released at https://huggingface.co/datasets/KuaishouHAIC/HAIC.
A two-stage data annotation pipeline enhances video understanding involving human actions by curating HAICTrain and HAICBench datasets, improving performance in human understanding and text-to-video generation.
Multi-modal Large Language Models video understanding human actions data annotation video-caption pairs human attributes text-to-video generation benchmarks