InstructExcel: A Benchmark for Natural Language Instruction in Excel
With the evolution of Large Language Models (LLMs) we can solve increasingly
more complex NLP tasks across various domains, including spreadsheets. This
work investigates whether LLMs can generate code (Excel OfficeScripts, a
TypeScript API for executing many tasks in Excel) that solves Excel specific
tasks provided via natural language user instructions. To do so we introduce a
new large-scale benchmark, InstructExcel, created by leveraging the 'Automate'
feature in Excel to automatically generate OfficeScripts from users' actions.
Our benchmark includes over 10k samples covering 170+ Excel operations across
2,000 publicly available Excel spreadsheets. Experiments across various
zero-shot and few-shot settings show that InstructExcel is a hard benchmark for
state of the art models like GPT-4. We observe that (1) using GPT-4 over
GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can
help improve performance on this benchmark.
65372bc28ee17cfd4406ebb0
Justin Payan
65372bc28ee17cfd4406ebb1
Swaroop Mishra
admin_assigned
2023-10-24T09:04:56.094Z
61841feb7c380c66d40cad54
/avatars/7a79362a24b844860781fb0625527eae.svg
Swaroop Mishra
user
swaroop7
65372bc28ee17cfd4406ebb2
Mukul Singh
claimed_verified
2023-10-24T14:27:28.839Z
63ea298fb1d9c4ef71eab282
/avatars/6a0d3b1678e2cf6b0356d0495530e045.svg
Mukul Singh
user
MukulSingh105
65372bc28ee17cfd4406ebb3
Carina Negreanu
65372bc28ee17cfd4406ebb4
Christian Poelitz
65372bc28ee17cfd4406ebb5
Chitta Baral
admin_assigned
2023-10-24T09:05:37.074Z
62687430dc80d589a9f6cd45
/avatars/fe56b8c92657e4f210c86cc097d42813.svg
Chitta Baral
user
cogint-asu
65372bc28ee17cfd4406ebb6
Subhro Roy
65372bc28ee17cfd4406ebb7
Rasika Chakravarthy
65372bc28ee17cfd4406ebb8
Benjamin Van Durme
65372bc28ee17cfd4406ebb9
Elnaz Nouri
InstructExcel: A Benchmark for Natural Language Instruction in Excel
With the evolution of Large Language Models (LLMs) we can solve increasingly
more complex NLP tasks across various domains, including spreadsheets. This
work investigates whether LLMs can generate code (Excel OfficeScripts, a
TypeScript API for executing many tasks in Excel) that solves Excel specific
tasks provided via natural language user instructions. To do so we introduce a
new large-scale benchmark, InstructExcel, created by leveraging the 'Automate'
feature in Excel to automatically generate OfficeScripts from users' actions.
Our benchmark includes over 10k samples covering 170+ Excel operations across
2,000 publicly available Excel spreadsheets. Experiments across various
zero-shot and few-shot settings show that InstructExcel is a hard benchmark for
state of the art models like GPT-4. We observe that (1) using GPT-4 over
GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can
help improve performance on this benchmark.
InstructExcel is a large-scale benchmark for evaluating LLMs' ability to generate Excel OfficeScripts from natural language instructions, demonstrating its difficulty for state-of-the-art models and identifying strategies to improve performance.
Large Language Models LLMs NLP tasks spreadsheets Excel OfficeScripts TypeScript API benchmark InstructExcel 'Automate' feature zero-shot few-shot settings GPT-4 GPT-3.5 dynamic prompting