15M Multimodal Facial Image-Text Dataset
<|reference_start|>15M Multimodal Facial Image-Text Dataset: Currently, image-text-driven multi-modal deep learning models have demonstrated their outstanding potential in many fields. In practice, tasks centered around facial images have broad application prospects. This paper presents \textbf{FaceCaption-15M}, a large-scale, diverse, and high-quality dataset of facial images accompanied by their natural language descriptions (facial image-to-text). This dataset aims to facilitate a study on face-centered tasks. FaceCaption-15M comprises over 15 million pairs of facial images and their corresponding natural language descriptions of facial features, making it the largest facial image-caption dataset to date. We conducted a comprehensive analysis of image quality, text naturalness, text complexity, and text-image relevance to demonstrate the superiority of FaceCaption-15M. To validate the effectiveness of FaceCaption-15M, we first trained a facial language-image pre-training model (FLIP, similar to CLIP) to align facial image with its corresponding captions in feature space. Subsequently, using both image and text encoders and fine-tuning only the linear layer, our FLIP-based models achieved state-of-the-art results on two challenging face-centered tasks. The purpose is to promote research in the field of face-related tasks through the availability of the proposed FaceCaption-15M dataset. All data, codes, and models are publicly available. https://huggingface.co/datasets/OpenFace-CQUPT/FaceCaption-15M<|reference_end|>
@article{dai202415m,
    title={15M Multimodal Facial Image-Text Dataset},
    author={Dawei Dai, YuTang Li, YingGe Liu, Mingming Jia, Zhang YuanHui, and
  Guoyin Wang},
    journal={arXiv preprint arXiv:2407.08515},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.08515},
    primaryClass={cs.CV cs.AI}
}