LLMs as Workers in Human-Computational Algorithms? Replicating
  Crowdsourcing Pipelines with LLMs
LLMs have shown promise in replicating human-like behavior in crowdsourcing
tasks that were previously thought to be exclusive to human abilities. However,
current efforts focus mainly on simple atomic tasks. We explore whether LLMs
can replicate more complex crowdsourcing pipelines. We find that modern LLMs
can simulate some of crowdworkers' abilities in these "human computation
algorithms," but the level of success is variable and influenced by requesters'
understanding of LLM capabilities, the specific skills required for sub-tasks,
and the optimal interaction modality for performing these sub-tasks. We reflect
on human and LLMs' different sensitivities to instructions, stress the
importance of enabling human-facing safeguards for LLMs, and discuss the
potential of training humans and LLMs with complementary skill sets. Crucially,
we show that replicating crowdsourcing pipelines offers a valuable platform to
investigate (1) the relative strengths of LLMs on different tasks (by
cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
complex tasks, where they can complete part of the tasks while leaving others
to humans.
64b8bf60d6ced0fd74b75978
Tongshuang Wu
admin_assigned
2023-07-20T07:45:38.942Z
5fe912eb8a3201f8e554c965
/avatars/7443e902d6febf4378eeee795a99ff60.svg
Tongshuang Wu
user
tongshuangwu
64b8bf60d6ced0fd74b75979
Haiyi Zhu
64b8bf60d6ced0fd74b7597a
Maya Albayrak
64b8bf60d6ced0fd74b7597b
Alexis Axon
64b8bf60d6ced0fd74b7597c
Amanda Bertsch
admin_assigned
2023-07-20T07:48:56.394Z
63a4c61fcde2b28f82a3baab
/avatars/dacd7792a37b4abf1ef67cd4a0312924.svg
Amanda Bertsch
user
abertsch
64b8bf60d6ced0fd74b7597d
Wenxing Deng
64b8bf60d6ced0fd74b7597e
Ziqi Ding
64b8bf60d6ced0fd74b7597f
Bill Guo
claimed_verified
2023-07-21T07:36:09.611Z
64b953d67ac999b4a162c79e
/avatars/987764e9d69dd4224483a43ae6424c0b.svg
Bill Guo
user
Duckbill
64b8bf60d6ced0fd74b75980
Sireesh Gururaja
admin_assigned
2023-07-20T07:50:12.128Z
62ed6b9d3d88d075d7253e13
https://cdn-avatars.huggingface.co/v1/production/uploads/62ed6b9d3d88d075d7253e13/pZeEP2pXdu78U6Jou529d.jpeg
Sireesh Gururaja
user
gsireesh
64b8bf60d6ced0fd74b75981
Tzu-Sheng Kuo
64b8bf60d6ced0fd74b75982
Jenny T. Liang
64b8bf60d6ced0fd74b75983
Ryan Liu
claimed_verified
2023-07-20T14:13:43.101Z
647a12028d17fcf64761a3fc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3GEiY7YCU_FzJing5wP-C.jpeg
Ryan Liu
user
theryanliu
64b8bf60d6ced0fd74b75984
Ihita Mandal
64b8bf60d6ced0fd74b75985
Jeremiah Milbauer
64b8bf60d6ced0fd74b75986
Xiaolin Ni
64b8bf60d6ced0fd74b75987
Namrata Padmanabhan
64b8bf60d6ced0fd74b75988
Subhashini Ramkumar
64b8bf60d6ced0fd74b75989
Alexis Sudjianto
64b8bf60d6ced0fd74b7598a
Jordan Taylor
claimed_verified
2023-07-21T07:35:59.026Z
64b9cf25be8a55986c918d8c
/avatars/7d78579b8781b78133f331e4cf56ed17.svg
Jordan
user
jordanta
64b8bf60d6ced0fd74b7598b
Ying-Jui Tseng
64b8bf60d6ced0fd74b7598c
Patricia Vaidos
claimed_verified
2023-07-21T07:36:06.289Z
64b987a5d4463c3d21814a94
/avatars/36964ad27d78008c14bd48d69f054832.svg
Pat Vaidos
user
pvaidos
64b8bf60d6ced0fd74b7598d
Zhijin Wu
64b8bf60d6ced0fd74b7598e
Wei Wu
64b8bf60d6ced0fd74b7598f
Chenyang Yang
claimed_verified
2023-07-21T07:36:02.870Z
6372ed8cbd1595ae66a5bbae
/avatars/21442e2c200eb015027215f0a06f7a8c.svg
Chenyang Yang
user
malusamayo
LLMs as Workers in Human-Computational Algorithms? Replicating
  Crowdsourcing Pipelines with LLMs
LLMs have shown promise in replicating human-like behavior in crowdsourcing
tasks that were previously thought to be exclusive to human abilities. However,
current efforts focus mainly on simple atomic tasks. We explore whether LLMs
can replicate more complex crowdsourcing pipelines. We find that modern LLMs
can simulate some of crowdworkers' abilities in these "human computation
algorithms," but the level of success is variable and influenced by requesters'
understanding of LLM capabilities, the specific skills required for sub-tasks,
and the optimal interaction modality for performing these sub-tasks. We reflect
on human and LLMs' different sensitivities to instructions, stress the
importance of enabling human-facing safeguards for LLMs, and discuss the
potential of training humans and LLMs with complementary skill sets. Crucially,
we show that replicating crowdsourcing pipelines offers a valuable platform to
investigate (1) the relative strengths of LLMs on different tasks (by
cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
complex tasks, where they can complete part of the tasks while leaving others
to humans.
Modern LLMs can simulate some aspects of complex crowdsourcing pipelines but their success varies based on task specifics, instruction clarity, and interaction modality.