title: LLMs as Workers in Human-Computational Algorithms? Replicating
    Crowdsourcing Pipelines with LLMs
summary: LLMs have shown promise in replicating human-like behavior in crowdsourcing
  tasks that were previously thought to be exclusive to human abilities. However,
  current efforts focus mainly on simple atomic tasks. We explore whether LLMs
  can replicate more complex crowdsourcing pipelines. We find that modern LLMs
  can simulate some of crowdworkers' abilities in these "human computation
  algorithms," but the level of success is variable and influenced by requesters'
  understanding of LLM capabilities, the specific skills required for sub-tasks,
  and the optimal interaction modality for performing these sub-tasks. We reflect
  on human and LLMs' different sensitivities to instructions, stress the
  importance of enabling human-facing safeguards for LLMs, and discuss the
  potential of training humans and LLMs with complementary skill sets. Crucially,
  we show that replicating crowdsourcing pipelines offers a valuable platform to
  investigate (1) the relative strengths of LLMs on different tasks (by
  cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
  complex tasks, where they can complete part of the tasks while leaving others
  to humans.
paper_authors._id: 64b8bf60d6ced0fd74b75978
paper_authors.name: Tongshuang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-20T07:45:38.942Z
paper_authors._id: 5fe912eb8a3201f8e554c965
paper_authors.avatarUrl: /avatars/7443e902d6febf4378eeee795a99ff60.svg
paper_authors.fullname: Tongshuang Wu
paper_authors.type: user
paper_authors.user: tongshuangwu
paper_authors._id: 64b8bf60d6ced0fd74b75979
paper_authors.name: Haiyi Zhu
paper_authors._id: 64b8bf60d6ced0fd74b7597a
paper_authors.name: Maya Albayrak
paper_authors._id: 64b8bf60d6ced0fd74b7597b
paper_authors.name: Alexis Axon
paper_authors._id: 64b8bf60d6ced0fd74b7597c
paper_authors.name: Amanda Bertsch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-20T07:48:56.394Z
paper_authors._id: 63a4c61fcde2b28f82a3baab
paper_authors.avatarUrl: /avatars/dacd7792a37b4abf1ef67cd4a0312924.svg
paper_authors.fullname: Amanda Bertsch
paper_authors.type: user
paper_authors.user: abertsch
paper_authors._id: 64b8bf60d6ced0fd74b7597d
paper_authors.name: Wenxing Deng
paper_authors._id: 64b8bf60d6ced0fd74b7597e
paper_authors.name: Ziqi Ding
paper_authors._id: 64b8bf60d6ced0fd74b7597f
paper_authors.name: Bill Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T07:36:09.611Z
paper_authors._id: 64b953d67ac999b4a162c79e
paper_authors.avatarUrl: /avatars/987764e9d69dd4224483a43ae6424c0b.svg
paper_authors.fullname: Bill Guo
paper_authors.type: user
paper_authors.user: Duckbill
paper_authors._id: 64b8bf60d6ced0fd74b75980
paper_authors.name: Sireesh Gururaja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-20T07:50:12.128Z
paper_authors._id: 62ed6b9d3d88d075d7253e13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ed6b9d3d88d075d7253e13/pZeEP2pXdu78U6Jou529d.jpeg
paper_authors.fullname: Sireesh Gururaja
paper_authors.type: user
paper_authors.user: gsireesh
paper_authors._id: 64b8bf60d6ced0fd74b75981
paper_authors.name: Tzu-Sheng Kuo
paper_authors._id: 64b8bf60d6ced0fd74b75982
paper_authors.name: Jenny T. Liang
paper_authors._id: 64b8bf60d6ced0fd74b75983
paper_authors.name: Ryan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-20T14:13:43.101Z
paper_authors._id: 647a12028d17fcf64761a3fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3GEiY7YCU_FzJing5wP-C.jpeg
paper_authors.fullname: Ryan Liu
paper_authors.type: user
paper_authors.user: theryanliu
paper_authors._id: 64b8bf60d6ced0fd74b75984
paper_authors.name: Ihita Mandal
paper_authors._id: 64b8bf60d6ced0fd74b75985
paper_authors.name: Jeremiah Milbauer
paper_authors._id: 64b8bf60d6ced0fd74b75986
paper_authors.name: Xiaolin Ni
paper_authors._id: 64b8bf60d6ced0fd74b75987
paper_authors.name: Namrata Padmanabhan
paper_authors._id: 64b8bf60d6ced0fd74b75988
paper_authors.name: Subhashini Ramkumar
paper_authors._id: 64b8bf60d6ced0fd74b75989
paper_authors.name: Alexis Sudjianto
paper_authors._id: 64b8bf60d6ced0fd74b7598a
paper_authors.name: Jordan Taylor
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T07:35:59.026Z
paper_authors._id: 64b9cf25be8a55986c918d8c
paper_authors.avatarUrl: /avatars/7d78579b8781b78133f331e4cf56ed17.svg
paper_authors.fullname: Jordan
paper_authors.type: user
paper_authors.user: jordanta
paper_authors._id: 64b8bf60d6ced0fd74b7598b
paper_authors.name: Ying-Jui Tseng
paper_authors._id: 64b8bf60d6ced0fd74b7598c
paper_authors.name: Patricia Vaidos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T07:36:06.289Z
paper_authors._id: 64b987a5d4463c3d21814a94
paper_authors.avatarUrl: /avatars/36964ad27d78008c14bd48d69f054832.svg
paper_authors.fullname: Pat Vaidos
paper_authors.type: user
paper_authors.user: pvaidos
paper_authors._id: 64b8bf60d6ced0fd74b7598d
paper_authors.name: Zhijin Wu
paper_authors._id: 64b8bf60d6ced0fd74b7598e
paper_authors.name: Wei Wu
paper_authors._id: 64b8bf60d6ced0fd74b7598f
paper_authors.name: Chenyang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T07:36:02.870Z
paper_authors._id: 6372ed8cbd1595ae66a5bbae
paper_authors.avatarUrl: /avatars/21442e2c200eb015027215f0a06f7a8c.svg
paper_authors.fullname: Chenyang Yang
paper_authors.type: user
paper_authors.user: malusamayo
paper_title: LLMs as Workers in Human-Computational Algorithms? Replicating
    Crowdsourcing Pipelines with LLMs
paper_summary: LLMs have shown promise in replicating human-like behavior in crowdsourcing
  tasks that were previously thought to be exclusive to human abilities. However,
  current efforts focus mainly on simple atomic tasks. We explore whether LLMs
  can replicate more complex crowdsourcing pipelines. We find that modern LLMs
  can simulate some of crowdworkers' abilities in these "human computation
  algorithms," but the level of success is variable and influenced by requesters'
  understanding of LLM capabilities, the specific skills required for sub-tasks,
  and the optimal interaction modality for performing these sub-tasks. We reflect
  on human and LLMs' different sensitivities to instructions, stress the
  importance of enabling human-facing safeguards for LLMs, and discuss the
  potential of training humans and LLMs with complementary skill sets. Crucially,
  we show that replicating crowdsourcing pipelines offers a valuable platform to
  investigate (1) the relative strengths of LLMs on different tasks (by
  cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
  complex tasks, where they can complete part of the tasks while leaving others
  to humans.
paper_ai_summary: Modern LLMs can simulate some aspects of complex crowdsourcing pipelines but their success varies based on task specifics, instruction clarity, and interaction modality.