$\textttACCORD$: Closing the Commonsense Measurability Gap
<|reference_start|>$\textttACCORD$: Closing the Commonsense Measurability Gap: We present $\texttt{ACCORD}$, a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. $\texttt{ACCORD}$ introduces formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. Uniquely, $\texttt{ACCORD}$ can automatically generate benchmarks of arbitrary reasoning complexity, and so it scales with future LLM improvements. Benchmarking state-of-the-art LLMs -- including GPT-4o (2024-05-13), Llama-3-70B-Instruct, and Mixtral-8x22B-Instruct-v0.1 -- shows performance degrading to random chance with only moderate scaling, leaving substantial headroom for improvement. We release a leaderboard of the benchmark suite tested in this work, as well as code for automatically generating more complex benchmarks.<|reference_end|>
@article{roewer-després2024$\texttt{accord}$:,
    title={$\texttt{ACCORD}$: Closing the Commonsense Measurability Gap},
    author={Franc{c}ois Roewer-Despr'es and Jinyue Feng and Zining Zhu and Frank
  Rudzicz},
    journal={arXiv preprint arXiv:2406.02804},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02804},
    primaryClass={cs.AI cs.CL cs.LG}
}