AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large
  Language Models
Verbatim feedback constitutes a valuable repository of user experiences,
opinions, and requirements essential for software development. Effectively and
efficiently extracting valuable insights from such data poses a challenging
task. This paper introduces Allhands , an innovative analytic framework
designed for large-scale feedback analysis through a natural language
interface, leveraging large language models (LLMs). Allhands adheres to a
conventional feedback analytic workflow, initially conducting classification
and topic modeling on the feedback to convert them into a structurally
augmented format, incorporating LLMs to enhance accuracy, robustness,
generalization, and user-friendliness. Subsequently, an LLM agent is employed
to interpret users' diverse questions in natural language on feedback,
translating them into Python code for execution, and delivering comprehensive
multi-modal responses, including text, code, tables, and images.
  We evaluate Allhands across three diverse feedback datasets. The experiments
demonstrate that Allhands achieves superior efficacy at all stages of analysis,
including classification and topic modeling, eventually providing users with an
``ask me anything'' experience with comprehensive, correct and human-readable
response. To the best of our knowledge, Allhands stands as the first
comprehensive feedback analysis framework that supports diverse and customized
requirements for insight extraction through a natural language interface.
6601032c003d5f16a9022dd0
Chaoyun Zhang
admin_assigned
2024-03-25T11:10:35.156Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
6601032c003d5f16a9022dd1
Zicheng Ma
6601032c003d5f16a9022dd2
Yuhao Wu
6601032c003d5f16a9022dd3
Shilin He
admin_assigned
2024-03-25T11:11:30.662Z
62c6df026a092eda1f1ab6e5
/avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
Shilin He
user
shilhe
6601032c003d5f16a9022dd4
Si Qin
6601032c003d5f16a9022dd5
Minghua Ma
admin_assigned
2024-03-25T11:11:56.365Z
653b52098abd634b83f4fd38
/avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
MingHua Ma
user
Gezelligheid520
6601032c003d5f16a9022dd6
Xiaoting Qin
6601032c003d5f16a9022dd7
Yu Kang
6601032c003d5f16a9022dd8
Yuyi Liang
6601032c003d5f16a9022dd9
Xiaoyu Gou
6601032c003d5f16a9022dda
Yajie Xue
6601032c003d5f16a9022ddb
Qingwei Lin
6601032c003d5f16a9022ddc
Saravan Rajmohan
6601032c003d5f16a9022ddd
Dongmei Zhang
6601032c003d5f16a9022dde
Qi Zhang
AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large
  Language Models
Verbatim feedback constitutes a valuable repository of user experiences,
opinions, and requirements essential for software development. Effectively and
efficiently extracting valuable insights from such data poses a challenging
task. This paper introduces Allhands , an innovative analytic framework
designed for large-scale feedback analysis through a natural language
interface, leveraging large language models (LLMs). Allhands adheres to a
conventional feedback analytic workflow, initially conducting classification
and topic modeling on the feedback to convert them into a structurally
augmented format, incorporating LLMs to enhance accuracy, robustness,
generalization, and user-friendliness. Subsequently, an LLM agent is employed
to interpret users' diverse questions in natural language on feedback,
translating them into Python code for execution, and delivering comprehensive
multi-modal responses, including text, code, tables, and images.
  We evaluate Allhands across three diverse feedback datasets. The experiments
demonstrate that Allhands achieves superior efficacy at all stages of analysis,
including classification and topic modeling, eventually providing users with an
``ask me anything'' experience with comprehensive, correct and human-readable
response. To the best of our knowledge, Allhands stands as the first
comprehensive feedback analysis framework that supports diverse and customized
requirements for insight extraction through a natural language interface.
Allhands is an innovative feedback analysis framework using large language models for classification, topic modeling, and query interpretation to deliver comprehensive insights via a natural language interface.
large language models natural language interface LLMs LLM agent feedback analysis classification topic modeling text code tables images