Towards Unified Facial Action Unit Recognition Framework by Large Language Models
<|reference_start|>Towards Unified Facial Action Unit Recognition Framework by Large Language Models: Facial Action Units (AUs) are of great significance in the realm of affective computing. In this paper, we propose AU-LLaVA, the first unified AU recognition framework based on the Large Language Model (LLM). AU-LLaVA consists of a visual encoder, a linear projector layer, and a pre-trained LLM. We meticulously craft the text descriptions and fine-tune the model on various AU datasets, allowing it to generate different formats of AU recognition results for the same input image. On the BP4D and DISFA datasets, AU-LLaVA delivers the most accurate recognition results for nearly half of the AUs. Our model achieves improvements of F1-score up to 11.4% in specific AU recognition compared to previous benchmark results. On the FEAFA dataset, our method achieves significant improvements over all 24 AUs compared to previous benchmark results. AU-LLaVA demonstrates exceptional performance and versatility in AU recognition.<|reference_end|>
@article{hu2024towards,
    title={Towards Unified Facial Action Unit Recognition Framework by Large
  Language Models},
    author={Guohong Hu, Xing Lan, Hanyu Jiang, Jiayi Lyu and Jian Xue},
    journal={arXiv preprint arXiv:2409.08444},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.08444},
    primaryClass={cs.CV}
}