Keypoints-Integrated Instruction-Following Data Generation for Enhanced Human Pose Understanding in Multimodal Models
<|reference_start|>Keypoints-Integrated Instruction-Following Data Generation for Enhanced Human Pose Understanding in Multimodal Models: Current multimodal models are well-suited for general visual understanding tasks. However, they perform inadequately when handling complex visual tasks related to human poses and actions, primarily due to the lack of specialized instruction-following data. We introduce a new method for generating such data by integrating human keypoints with traditional visual features like captions and bounding boxes. Our approach produces datasets designed for fine-tuning models to excel in human-centric activities, focusing on three specific types: conversation, detailed description, and complex reasoning. We fine-tuned the LLaVA-7B model with this novel dataset, achieving significant improvements across various human pose-related tasks. Experimental results show an overall improvement of 21.18% compared to the original LLaVA-7B model. These findings demonstrate the effectiveness of keypoints-assisted data in enhancing multimodal models.<|reference_end|>
@article{zhang2024keypoints-integrated,
    title={Keypoints-Integrated Instruction-Following Data Generation for Enhanced
  Human Pose Understanding in Multimodal Models},
    author={Dewen Zhang, Wangpeng An, Hayaru Shouno},
    journal={arXiv preprint arXiv:2409.09306},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09306},
    primaryClass={cs.CV}
}