title: The SIFo Benchmark: Investigating the Sequential Instruction Following
    Ability of Large Language Models
summary: Following multiple instructions is a crucial ability for large language
  models (LLMs). Evaluating this ability comes with significant challenges: (i)
  limited coherence between multiple instructions, (ii) positional bias where the
  order of instructions affects model performance, and (iii) a lack of
  objectively verifiable tasks. To address these issues, we introduce a benchmark
  designed to evaluate models' abilities to follow multiple instructions through
  sequential instruction following (SIFo) tasks. In SIFo, the successful
  completion of multiple instructions is verifiable by examining only the final
  instruction. Our benchmark evaluates instruction following using four tasks
  (text modification, question answering, mathematics, and security rule
  following), each assessing different aspects of sequential instruction
  following. Our evaluation of popular LLMs, both closed-source and open-source,
  shows that more recent and larger models significantly outperform their older
  and smaller counterparts on the SIFo tasks, validating the benchmark's
  effectiveness. All models struggle with following sequences of instructions,
  hinting at an important lack of robustness of today's language models.
paper_authors._id: 6684902742cbc4ce0b464bd8
paper_authors.name: Xinyi Chen
paper_authors._id: 6684902742cbc4ce0b464bd9
paper_authors.name: Baohao Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:07:56.891Z
paper_authors._id: 62c414354ce7250560a1f67f
paper_authors.avatarUrl: /avatars/28fd73973d1703c84f4f59644fef8a80.svg
paper_authors.fullname: Baohao Liao
paper_authors.type: user
paper_authors.user: baohao
paper_authors._id: 6684902742cbc4ce0b464bda
paper_authors.name: Jirui Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T07:38:09.515Z
paper_authors._id: 645a415077d040cfd64e036e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645a415077d040cfd64e036e/2k6QsstgWC1H4oRMlzcc5.png
paper_authors.fullname: Jirui Qi
paper_authors.type: user
paper_authors.user: JRQi
paper_authors._id: 6684902742cbc4ce0b464bdb
paper_authors.name: Panagiotis Eustratiadis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:07:35.227Z
paper_authors._id: 651fcc8c5d91101181e14b8f
paper_authors.avatarUrl: /avatars/6523f432859065465cf30d1143a2ce3e.svg
paper_authors.fullname: Panagiotis Eustratiadis
paper_authors.type: user
paper_authors.user: peustr
paper_authors._id: 6684902742cbc4ce0b464bdc
paper_authors.name: Christof Monz
paper_authors._id: 6684902742cbc4ce0b464bdd
paper_authors.name: Arianna Bisazza
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:07:46.952Z
paper_authors._id: 61e050c73a1781f66b4e9ab6
paper_authors.avatarUrl: /avatars/9c91a18cdc53587422311fd13a14833e.svg
paper_authors.fullname: Arianna Bisazza
paper_authors.type: user
paper_authors.user: arianna-bis
paper_authors._id: 6684902742cbc4ce0b464bde
paper_authors.name: Maarten de Rijke
paper_title: The SIFo Benchmark: Investigating the Sequential Instruction Following
    Ability of Large Language Models
paper_summary: Following multiple instructions is a crucial ability for large language
  models (LLMs). Evaluating this ability comes with significant challenges: (i)
  limited coherence between multiple instructions, (ii) positional bias where the
  order of instructions affects model performance, and (iii) a lack of
  objectively verifiable tasks. To address these issues, we introduce a benchmark
  designed to evaluate models' abilities to follow multiple instructions through
  sequential instruction following (SIFo) tasks. In SIFo, the successful
  completion of multiple instructions is verifiable by examining only the final
  instruction. Our benchmark evaluates instruction following using four tasks
  (text modification, question answering, mathematics, and security rule
  following), each assessing different aspects of sequential instruction
  following. Our evaluation of popular LLMs, both closed-source and open-source,
  shows that more recent and larger models significantly outperform their older
  and smaller counterparts on the SIFo tasks, validating the benchmark's
  effectiveness. All models struggle with following sequences of instructions,
  hinting at an important lack of robustness of today's language models.
paper_ai_summary: A benchmark evaluates large language models' ability to follow multiple instructions through sequential instruction following tasks, revealing performance gaps between new and old models.
paper_ai_keywords: large language models sequential instruction following SIFo text modification question answering mathematics security rule following