title: AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large
    Language Models
summary: Verbatim feedback constitutes a valuable repository of user experiences,
  opinions, and requirements essential for software development. Effectively and
  efficiently extracting valuable insights from such data poses a challenging
  task. This paper introduces Allhands , an innovative analytic framework
  designed for large-scale feedback analysis through a natural language
  interface, leveraging large language models (LLMs). Allhands adheres to a
  conventional feedback analytic workflow, initially conducting classification
  and topic modeling on the feedback to convert them into a structurally
  augmented format, incorporating LLMs to enhance accuracy, robustness,
  generalization, and user-friendliness. Subsequently, an LLM agent is employed
  to interpret users' diverse questions in natural language on feedback,
  translating them into Python code for execution, and delivering comprehensive
  multi-modal responses, including text, code, tables, and images.
    We evaluate Allhands across three diverse feedback datasets. The experiments
  demonstrate that Allhands achieves superior efficacy at all stages of analysis,
  including classification and topic modeling, eventually providing users with an
  ``ask me anything'' experience with comprehensive, correct and human-readable
  response. To the best of our knowledge, Allhands stands as the first
  comprehensive feedback analysis framework that supports diverse and customized
  requirements for insight extraction through a natural language interface.
paper_authors._id: 6601032c003d5f16a9022dd0
paper_authors.name: Chaoyun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:10:35.156Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 6601032c003d5f16a9022dd1
paper_authors.name: Zicheng Ma
paper_authors._id: 6601032c003d5f16a9022dd2
paper_authors.name: Yuhao Wu
paper_authors._id: 6601032c003d5f16a9022dd3
paper_authors.name: Shilin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:11:30.662Z
paper_authors._id: 62c6df026a092eda1f1ab6e5
paper_authors.avatarUrl: /avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
paper_authors.fullname: Shilin He
paper_authors.type: user
paper_authors.user: shilhe
paper_authors._id: 6601032c003d5f16a9022dd4
paper_authors.name: Si Qin
paper_authors._id: 6601032c003d5f16a9022dd5
paper_authors.name: Minghua Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T11:11:56.365Z
paper_authors._id: 653b52098abd634b83f4fd38
paper_authors.avatarUrl: /avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
paper_authors.fullname: MingHua Ma
paper_authors.type: user
paper_authors.user: Gezelligheid520
paper_authors._id: 6601032c003d5f16a9022dd6
paper_authors.name: Xiaoting Qin
paper_authors._id: 6601032c003d5f16a9022dd7
paper_authors.name: Yu Kang
paper_authors._id: 6601032c003d5f16a9022dd8
paper_authors.name: Yuyi Liang
paper_authors._id: 6601032c003d5f16a9022dd9
paper_authors.name: Xiaoyu Gou
paper_authors._id: 6601032c003d5f16a9022dda
paper_authors.name: Yajie Xue
paper_authors._id: 6601032c003d5f16a9022ddb
paper_authors.name: Qingwei Lin
paper_authors._id: 6601032c003d5f16a9022ddc
paper_authors.name: Saravan Rajmohan
paper_authors._id: 6601032c003d5f16a9022ddd
paper_authors.name: Dongmei Zhang
paper_authors._id: 6601032c003d5f16a9022dde
paper_authors.name: Qi Zhang
paper_title: AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large
    Language Models
paper_summary: Verbatim feedback constitutes a valuable repository of user experiences,
  opinions, and requirements essential for software development. Effectively and
  efficiently extracting valuable insights from such data poses a challenging
  task. This paper introduces Allhands , an innovative analytic framework
  designed for large-scale feedback analysis through a natural language
  interface, leveraging large language models (LLMs). Allhands adheres to a
  conventional feedback analytic workflow, initially conducting classification
  and topic modeling on the feedback to convert them into a structurally
  augmented format, incorporating LLMs to enhance accuracy, robustness,
  generalization, and user-friendliness. Subsequently, an LLM agent is employed
  to interpret users' diverse questions in natural language on feedback,
  translating them into Python code for execution, and delivering comprehensive
  multi-modal responses, including text, code, tables, and images.
    We evaluate Allhands across three diverse feedback datasets. The experiments
  demonstrate that Allhands achieves superior efficacy at all stages of analysis,
  including classification and topic modeling, eventually providing users with an
  ``ask me anything'' experience with comprehensive, correct and human-readable
  response. To the best of our knowledge, Allhands stands as the first
  comprehensive feedback analysis framework that supports diverse and customized
  requirements for insight extraction through a natural language interface.
paper_ai_keywords: large language models (LLMs) classification topic modeling natural language interface Python code execution multi-modal responses semi-modal responses text code tables images feedback datasets ``ask me anything'' experience human-readable responses