FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability
<|reference_start|>FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability: This paper presents FoFo, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats, a crucial yet underexamined capability for their application as AI agents. Despite LLMs' advancements, existing benchmarks fail to assess their format-following proficiency adequately. FoFo fills this gap with a diverse range of real-world formats and instructions, developed through an AI-Human collaborative method. Our evaluation across both open-source (e.g., Llama 2, WizardLM) and closed-source (e.g., GPT-4, PALM2, Gemini) LLMs highlights three key findings: open-source models significantly lag behind closed-source ones in format adherence; LLMs' format-following performance is independent of their content generation quality; and LLMs' format proficiency varies across different domains. These insights suggest the need for specialized tuning for format-following skills and highlight FoFo's role in guiding the selection of domain-specific AI agents. FoFo is released here at https://github.com/SalesforceAIResearch/FoFo.<|reference_end|>
@article{xia2024fofo:,
    title={FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability},
    author={Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu,
  Wenpeng Yin, Caiming Xiong},
    journal={arXiv preprint arXiv:2402.18667},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.18667},
    primaryClass={cs.CL}
}