Requirements Satisfiability with In-Context Learning
<|reference_start|>Requirements Satisfiability with In-Context Learning: Language models that can learn a task at inference time, called in-context learning (ICL), show increasing promise in natural language inference tasks. In ICL, a model user constructs a prompt to describe a task with a natural language instruction and zero or more examples, called demonstrations. The prompt is then input to the language model to generate a completion. In this paper, we apply ICL to the design and evaluation of satisfaction arguments, which describe how a requirement is satisfied by a system specification and associated domain knowledge. The approach builds on three prompt design patterns, including augmented generation, prompt tuning, and chain-of-thought prompting, and is evaluated on a privacy problem to check whether a mobile app scenario and associated design description satisfies eight consent requirements from the EU General Data Protection Regulation (GDPR). The overall results show that GPT-4 can be used to verify requirements satisfaction with 96.7% accuracy and dissatisfaction with 93.2% accuracy. Inverting the requirement improves verification of dissatisfaction to 97.2%. Chain-of-thought prompting improves overall GPT-3.5 performance by 9.0% accuracy. We discuss the trade-offs among templates, models and prompt strategies and provide a detailed analysis of the generated specifications to inform how the approach can be applied in practice.<|reference_end|>
@article{santos2024requirements,
    title={Requirements Satisfiability with In-Context Learning},
    author={Sarah Santos, Travis Breaux, Thomas Norton, Sara Haghighi, Sepideh
  Ghanavati},
    journal={IEEE International Requirements Engineering Conference, 2024},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.12576},
    primaryClass={cs.SE}
}