title: InstructExcel: A Benchmark for Natural Language Instruction in Excel
summary: With the evolution of Large Language Models (LLMs) we can solve increasingly
  more complex NLP tasks across various domains, including spreadsheets. This
  work investigates whether LLMs can generate code (Excel OfficeScripts, a
  TypeScript API for executing many tasks in Excel) that solves Excel specific
  tasks provided via natural language user instructions. To do so we introduce a
  new large-scale benchmark, InstructExcel, created by leveraging the 'Automate'
  feature in Excel to automatically generate OfficeScripts from users' actions.
  Our benchmark includes over 10k samples covering 170+ Excel operations across
  2,000 publicly available Excel spreadsheets. Experiments across various
  zero-shot and few-shot settings show that InstructExcel is a hard benchmark for
  state of the art models like GPT-4. We observe that (1) using GPT-4 over
  GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can
  help improve performance on this benchmark.
paper_authors._id: 65372bc28ee17cfd4406ebb0
paper_authors.name: Justin Payan
paper_authors._id: 65372bc28ee17cfd4406ebb1
paper_authors.name: Swaroop Mishra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T09:04:56.094Z
paper_authors._id: 61841feb7c380c66d40cad54
paper_authors.avatarUrl: /avatars/7a79362a24b844860781fb0625527eae.svg
paper_authors.fullname: Swaroop Mishra
paper_authors.type: user
paper_authors.user: swaroop7
paper_authors._id: 65372bc28ee17cfd4406ebb2
paper_authors.name: Mukul Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T14:27:28.839Z
paper_authors._id: 63ea298fb1d9c4ef71eab282
paper_authors.avatarUrl: /avatars/6a0d3b1678e2cf6b0356d0495530e045.svg
paper_authors.fullname: Mukul Singh
paper_authors.type: user
paper_authors.user: MukulSingh105
paper_authors._id: 65372bc28ee17cfd4406ebb3
paper_authors.name: Carina Negreanu
paper_authors._id: 65372bc28ee17cfd4406ebb4
paper_authors.name: Christian Poelitz
paper_authors._id: 65372bc28ee17cfd4406ebb5
paper_authors.name: Chitta Baral
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T09:05:37.074Z
paper_authors._id: 62687430dc80d589a9f6cd45
paper_authors.avatarUrl: /avatars/fe56b8c92657e4f210c86cc097d42813.svg
paper_authors.fullname: Chitta Baral
paper_authors.type: user
paper_authors.user: cogint-asu
paper_authors._id: 65372bc28ee17cfd4406ebb6
paper_authors.name: Subhro Roy
paper_authors._id: 65372bc28ee17cfd4406ebb7
paper_authors.name: Rasika Chakravarthy
paper_authors._id: 65372bc28ee17cfd4406ebb8
paper_authors.name: Benjamin Van Durme
paper_authors._id: 65372bc28ee17cfd4406ebb9
paper_authors.name: Elnaz Nouri
paper_title: InstructExcel: A Benchmark for Natural Language Instruction in Excel
paper_summary: With the evolution of Large Language Models (LLMs) we can solve increasingly
  more complex NLP tasks across various domains, including spreadsheets. This
  work investigates whether LLMs can generate code (Excel OfficeScripts, a
  TypeScript API for executing many tasks in Excel) that solves Excel specific
  tasks provided via natural language user instructions. To do so we introduce a
  new large-scale benchmark, InstructExcel, created by leveraging the 'Automate'
  feature in Excel to automatically generate OfficeScripts from users' actions.
  Our benchmark includes over 10k samples covering 170+ Excel operations across
  2,000 publicly available Excel spreadsheets. Experiments across various
  zero-shot and few-shot settings show that InstructExcel is a hard benchmark for
  state of the art models like GPT-4. We observe that (1) using GPT-4 over
  GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can
  help improve performance on this benchmark.
paper_ai_summary: InstructExcel is a large-scale benchmark for evaluating LLMs' ability to generate Excel OfficeScripts from natural language instructions, demonstrating its difficulty for state-of-the-art models and identifying strategies to improve performance.
paper_ai_keywords: Large Language Models LLMs NLP tasks spreadsheets Excel OfficeScripts TypeScript API benchmark InstructExcel 'Automate' feature zero-shot few-shot settings GPT-4 GPT-3.5 dynamic prompting