title: HAIC: Improving Human Action Understanding and Generation with Better
    Captions for Multi-modal Large Language Models
summary: Recent Multi-modal Large Language Models (MLLMs) have made great progress in
  video understanding. However, their performance on videos involving human
  actions is still limited by the lack of high-quality data. To address this, we
  introduce a two-stage data annotation pipeline. First, we design strategies to
  accumulate videos featuring clear human actions from the Internet. Second,
  videos are annotated in a standardized caption format that uses human
  attributes to distinguish individuals and chronologically details their actions
  and interactions. Through this pipeline, we curate two datasets, namely
  HAICTrain and HAICBench. HAICTrain comprises 126K video-caption pairs
  generated by Gemini-Pro and verified for training purposes. Meanwhile,
  HAICBench includes 500 manually annotated video-caption pairs and
  1,400 QA pairs, for a comprehensive evaluation of human action understanding.
  Experimental results demonstrate that training with HAICTrain not only
  significantly enhances human understanding abilities across 4 benchmarks, but
  can also improve text-to-video generation results. Both the HAICTrain and
  HAICBench are released at https://huggingface.co/datasets/KuaishouHAIC/HAIC.
paper_authors._id: 67c51c198d02783fa3a6249d
paper_authors.name: Xiao Wang
paper_authors._id: 67c51c198d02783fa3a6249e
paper_authors.name: Jingyun Hua
paper_authors._id: 67c51c198d02783fa3a6249f
paper_authors.name: Weihong Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-03T09:42:30.547Z
paper_authors._id: 675a69699e086bd6250a36ef
paper_authors.avatarUrl: /avatars/95c72e3975d1a37f8655a2fe629746ec.svg
paper_authors.fullname: Weihong Lin
paper_authors.type: user
paper_authors.user: lwher1996
paper_authors._id: 67c51c198d02783fa3a624a0
paper_authors.name: Yuanxing Zhang
paper_authors._id: 67c51c198d02783fa3a624a1
paper_authors.name: Fuzheng Zhang
paper_authors._id: 67c51c198d02783fa3a624a2
paper_authors.name: Jianlong Wu
paper_authors._id: 67c51c198d02783fa3a624a3
paper_authors.name: Di Zhang
paper_authors._id: 67c51c198d02783fa3a624a4
paper_authors.name: Liqiang Nie
paper_title: HAIC: Improving Human Action Understanding and Generation with Better
    Captions for Multi-modal Large Language Models
paper_summary: Recent Multi-modal Large Language Models (MLLMs) have made great progress in
  video understanding. However, their performance on videos involving human
  actions is still limited by the lack of high-quality data. To address this, we
  introduce a two-stage data annotation pipeline. First, we design strategies to
  accumulate videos featuring clear human actions from the Internet. Second,
  videos are annotated in a standardized caption format that uses human
  attributes to distinguish individuals and chronologically details their actions
  and interactions. Through this pipeline, we curate two datasets, namely
  HAICTrain and HAICBench. HAICTrain comprises 126K video-caption pairs
  generated by Gemini-Pro and verified for training purposes. Meanwhile,
  HAICBench includes 500 manually annotated video-caption pairs and
  1,400 QA pairs, for a comprehensive evaluation of human action understanding.
  Experimental results demonstrate that training with HAICTrain not only
  significantly enhances human understanding abilities across 4 benchmarks, but
  can also improve text-to-video generation results. Both the HAICTrain and
  HAICBench are released at https://huggingface.co/datasets/KuaishouHAIC/HAIC.