The SIFo Benchmark: Investigating the Sequential Instruction Following
  Ability of Large Language Models
Following multiple instructions is a crucial ability for large language
models (LLMs). Evaluating this ability comes with significant challenges: (i)
limited coherence between multiple instructions, (ii) positional bias where the
order of instructions affects model performance, and (iii) a lack of
objectively verifiable tasks. To address these issues, we introduce a benchmark
designed to evaluate models' abilities to follow multiple instructions through
sequential instruction following (SIFo) tasks. In SIFo, the successful
completion of multiple instructions is verifiable by examining only the final
instruction. Our benchmark evaluates instruction following using four tasks
(text modification, question answering, mathematics, and security rule
following), each assessing different aspects of sequential instruction
following. Our evaluation of popular LLMs, both closed-source and open-source,
shows that more recent and larger models significantly outperform their older
and smaller counterparts on the SIFo tasks, validating the benchmark's
effectiveness. All models struggle with following sequences of instructions,
hinting at an important lack of robustness of today's language models.
6684902742cbc4ce0b464bd8
Xinyi Chen
6684902742cbc4ce0b464bd9
Baohao Liao
admin_assigned
2024-07-03T15:07:56.891Z
62c414354ce7250560a1f67f
/avatars/28fd73973d1703c84f4f59644fef8a80.svg
Baohao Liao
user
baohao
6684902742cbc4ce0b464bda
Jirui Qi
claimed_verified
2024-07-03T07:38:09.515Z
645a415077d040cfd64e036e
https://cdn-avatars.huggingface.co/v1/production/uploads/645a415077d040cfd64e036e/2k6QsstgWC1H4oRMlzcc5.png
Jirui Qi
user
JRQi
6684902742cbc4ce0b464bdb
Panagiotis Eustratiadis
admin_assigned
2024-07-03T15:07:35.227Z
651fcc8c5d91101181e14b8f
/avatars/6523f432859065465cf30d1143a2ce3e.svg
Panagiotis Eustratiadis
user
peustr
6684902742cbc4ce0b464bdc
Christof Monz
6684902742cbc4ce0b464bdd
Arianna Bisazza
admin_assigned
2024-07-03T15:07:46.952Z
61e050c73a1781f66b4e9ab6
/avatars/9c91a18cdc53587422311fd13a14833e.svg
Arianna Bisazza
user
arianna-bis
6684902742cbc4ce0b464bde
Maarten de Rijke
The SIFo Benchmark: Investigating the Sequential Instruction Following
  Ability of Large Language Models
Following multiple instructions is a crucial ability for large language
models (LLMs). Evaluating this ability comes with significant challenges: (i)
limited coherence between multiple instructions, (ii) positional bias where the
order of instructions affects model performance, and (iii) a lack of
objectively verifiable tasks. To address these issues, we introduce a benchmark
designed to evaluate models' abilities to follow multiple instructions through
sequential instruction following (SIFo) tasks. In SIFo, the successful
completion of multiple instructions is verifiable by examining only the final
instruction. Our benchmark evaluates instruction following using four tasks
(text modification, question answering, mathematics, and security rule
following), each assessing different aspects of sequential instruction
following. Our evaluation of popular LLMs, both closed-source and open-source,
shows that more recent and larger models significantly outperform their older
and smaller counterparts on the SIFo tasks, validating the benchmark's
effectiveness. All models struggle with following sequences of instructions,
hinting at an important lack of robustness of today's language models.
A benchmark evaluates large language models' ability to follow multiple instructions through sequential instruction following tasks, revealing performance gaps between new and old models.
large language models sequential instruction following SIFo text modification question answering mathematics security rule following