title: Skill Discovery for Software Scripting Automation via Offline
    Simulations with LLMs
summary: Scripting interfaces enable users to automate tasks and customize software
  workflows, but creating scripts traditionally requires programming expertise
  and familiarity with specific APIs, posing barriers for many users. While Large
  Language Models (LLMs) can generate code from natural language queries, runtime
  code generation is severely limited due to unverified code, security risks,
  longer response times, and higher computational costs. To bridge the gap, we
  propose an offline simulation framework to curate a software-specific skillset,
  a collection of verified scripts, by exploiting LLMs and publicly available
  scripting guides. Our framework comprises two components: (1) task creation,
  using top-down functionality guidance and bottom-up API synergy exploration to
  generate helpful tasks; and (2) skill generation with trials, refining and
  validating scripts based on execution feedback. To efficiently navigate the
  extensive API landscape, we introduce a Graph Neural Network (GNN)-based link
  prediction model to capture API synergy, enabling the generation of skills
  involving underutilized APIs and expanding the skillset's diversity.
  Experiments with Adobe Illustrator demonstrate that our framework significantly
  improves automation success rates, reduces response time, and saves runtime
  token costs compared to traditional runtime code generation. This is the first
  attempt to use software scripting interfaces as a testbed for LLM-based
  systems, highlighting the advantages of leveraging execution feedback in a
  controlled environment and offering valuable insights into aligning AI
  capabilities with user needs in specialized software domains.
paper_authors._id: 681561d98a46db172ad08f35
paper_authors.name: Paiheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:49:56.029Z
paper_authors._id: 62505101a0f6b0ed18114323
paper_authors.avatarUrl: /avatars/fdd0cd6abba33740b037b71876e8af41.svg
paper_authors.fullname: Paiheng Xu
paper_authors.type: user
paper_authors.user: paiheng
paper_authors._id: 681561d98a46db172ad08f36
paper_authors.name: Gang Wu
paper_authors._id: 681561d98a46db172ad08f37
paper_authors.name: Xiang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:50:05.179Z
paper_authors._id: 64c073417255ff868721fb54
paper_authors.avatarUrl: /avatars/c2a00114598aa89f071b937f0af4077f.svg
paper_authors.fullname: Xiang CHEN
paper_authors.type: user
paper_authors.user: xiangchen-dvi
paper_authors._id: 681561d98a46db172ad08f38
paper_authors.name: Tong Yu
paper_authors._id: 681561d98a46db172ad08f39
paper_authors.name: Chang Xiao
paper_authors._id: 681561d98a46db172ad08f3a
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-04T10:04:28.382Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 681561d98a46db172ad08f3b
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:33:17.253Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 681561d98a46db172ad08f3c
paper_authors.name: Wei Ai
paper_authors._id: 681561d98a46db172ad08f3d
paper_authors.name: Viswanathan Swaminathan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:50:32.106Z
paper_authors._id: 66a443ed55a50dd32d4034e8
paper_authors.avatarUrl: /avatars/5a75cfa6137459f01309ecc38ea386cf.svg
paper_authors.fullname: Viswanathan Swaminathan
paper_authors.type: user
paper_authors.user: viswathegreat
paper_title: Skill Discovery for Software Scripting Automation via Offline
    Simulations with LLMs
paper_summary: Scripting interfaces enable users to automate tasks and customize software
  workflows, but creating scripts traditionally requires programming expertise
  and familiarity with specific APIs, posing barriers for many users. While Large
  Language Models (LLMs) can generate code from natural language queries, runtime
  code generation is severely limited due to unverified code, security risks,
  longer response times, and higher computational costs. To bridge the gap, we
  propose an offline simulation framework to curate a software-specific skillset,
  a collection of verified scripts, by exploiting LLMs and publicly available
  scripting guides. Our framework comprises two components: (1) task creation,
  using top-down functionality guidance and bottom-up API synergy exploration to
  generate helpful tasks; and (2) skill generation with trials, refining and
  validating scripts based on execution feedback. To efficiently navigate the
  extensive API landscape, we introduce a Graph Neural Network (GNN)-based link
  prediction model to capture API synergy, enabling the generation of skills
  involving underutilized APIs and expanding the skillset's diversity.
  Experiments with Adobe Illustrator demonstrate that our framework significantly
  improves automation success rates, reduces response time, and saves runtime
  token costs compared to traditional runtime code generation. This is the first
  attempt to use software scripting interfaces as a testbed for LLM-based
  systems, highlighting the advantages of leveraging execution feedback in a
  controlled environment and offering valuable insights into aligning AI
  capabilities with user needs in specialized software domains.
paper_ai_summary: An offline framework using LLMs and GNNs curates verified scripts for specific software APIs, improving automation success rates and reducing runtime costs.
paper_ai_keywords: Large Language Models runtime code generation Graph Neural Network link prediction model API synergy verified scripts task creation skill generation software scripting interfaces execution feedback Adobe Illustrator automation success rates runtime token costs