title: Can Large Language Models Transform Computational Social Science?
summary: Large Language Models (LLMs) like ChatGPT are capable of successfully
  performing many language processing tasks zero-shot (without the need for
  training data). If this capacity also applies to the coding of social phenomena
  like persuasiveness and political ideology, then LLMs could effectively
  transform Computational Social Science (CSS). This work provides a road map for
  using LLMs as CSS tools. Towards this end, we contribute a set of prompting
  best practices and an extensive evaluation pipeline to measure the zero-shot
  performance of 13 language models on 24 representative CSS benchmarks. On
  taxonomic labeling tasks (classification), LLMs fail to outperform the best
  fine-tuned models but still achieve fair levels of agreement with humans. On
  free-form coding tasks (generation), LLMs produce explanations that often
  exceed the quality of crowdworkers' gold references. We conclude that today's
  LLMs can radically augment the CSS research pipeline in two ways: (1) serving
  as zero-shot data annotators on human annotation teams, and (2) bootstrapping
  challenging creative generation tasks (e.g., explaining the hidden meaning
  behind text). In summary, LLMs can significantly reduce costs and increase
  efficiency of social science analysis in partnership with humans.
paper_authors._id: 6458461ef8bdc3512d1f5673
paper_authors.name: Caleb Ziems
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:58:09.891Z
paper_authors._id: 60b95db8337c67f46490d6bb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663610682205-60b95db8337c67f46490d6bb.jpeg
paper_authors.fullname: Caleb Ziems
paper_authors.type: user
paper_authors.user: cjziems
paper_authors._id: 6458461ef8bdc3512d1f5674
paper_authors.name: William Held
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T13:27:10.598Z
paper_authors._id: 632116accafe12f481a473cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666207676653-632116accafe12f481a473cb.jpeg
paper_authors.fullname: Will Held
paper_authors.type: user
paper_authors.user: WillHeld
paper_authors._id: 6458461ef8bdc3512d1f5675
paper_authors.name: Omar Shaikh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:58:40.538Z
paper_authors._id: 632601d42a0af3bd4522aed7
paper_authors.avatarUrl: /avatars/4c02961e3a944c026c6444094d4642aa.svg
paper_authors.fullname: Omar Shaikh
paper_authors.type: user
paper_authors.user: oshaikh13
paper_authors._id: 6458461ef8bdc3512d1f5676
paper_authors.name: Jiaao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:59:13.867Z
paper_authors._id: 6327c45946ffcfc27cfa1ee3
paper_authors.avatarUrl: /avatars/f64b36e978fc369bfdfebc16057cd67e.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Jiaao
paper_authors._id: 6458461ef8bdc3512d1f5677
paper_authors.name: Zhehao Zhang
paper_authors._id: 6458461ef8bdc3512d1f5678
paper_authors.name: Diyi Yang
paper_title: Can Large Language Models Transform Computational Social Science?
paper_summary: Large Language Models (LLMs) like ChatGPT are capable of successfully
  performing many language processing tasks zero-shot (without the need for
  training data). If this capacity also applies to the coding of social phenomena
  like persuasiveness and political ideology, then LLMs could effectively
  transform Computational Social Science (CSS). This work provides a road map for
  using LLMs as CSS tools. Towards this end, we contribute a set of prompting
  best practices and an extensive evaluation pipeline to measure the zero-shot
  performance of 13 language models on 24 representative CSS benchmarks. On
  taxonomic labeling tasks (classification), LLMs fail to outperform the best
  fine-tuned models but still achieve fair levels of agreement with humans. On
  free-form coding tasks (generation), LLMs produce explanations that often
  exceed the quality of crowdworkers' gold references. We conclude that today's
  LLMs can radically augment the CSS research pipeline in two ways: (1) serving
  as zero-shot data annotators on human annotation teams, and (2) bootstrapping
  challenging creative generation tasks (e.g., explaining the hidden meaning
  behind text). In summary, LLMs can significantly reduce costs and increase
  efficiency of social science analysis in partnership with humans.
paper_ai_summary: LLMs can enhance Computational Social Science by serving as zero-shot annotators and bootstrapping creative generation tasks, improving efficiency and reducing costs in social science analysis.
paper_ai_keywords: Large Language Models LLMs zero-shot performance prompting best practices evaluation pipeline CSS Computational Social Science taxonomic labeling tasks free-form coding tasks fine-tuned models data annotators human annotation teams creative generation tasks