title: Open-Source Large Language Models Outperform Crowd Workers and Approach
    ChatGPT in Text-Annotation Tasks
summary: This study examines the performance of open-source Large Language Models
  (LLMs) in text annotation tasks and compares it with proprietary models like
  ChatGPT and human-based services such as MTurk. While prior research
  demonstrated the high performance of ChatGPT across numerous NLP tasks,
  open-source LLMs like HugginChat and FLAN are gaining attention for their
  cost-effectiveness, transparency, reproducibility, and superior data
  protection. We assess these models using both zero-shot and few-shot approaches
  and different temperature parameters across a range of text annotation tasks.
  Our findings show that while ChatGPT achieves the best performance in most
  tasks, open-source LLMs not only outperform MTurk but also demonstrate
  competitive potential against ChatGPT in specific tasks.
paper_authors._id: 64a61ddbbec7a63923432da7
paper_authors.name: Meysam Alizadeh
paper_authors._id: 64a61ddbbec7a63923432da8
paper_authors.name: Maël Kubli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-06T14:42:57.481Z
paper_authors._id: 64a6cb90eb672ec8d53737c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a6cb90eb672ec8d53737c3/eEICiyB6q_Tv4BYTjOOx8.jpeg
paper_authors.fullname: Mael Kubli
paper_authors.type: user
paper_authors.user: kublima
paper_authors._id: 64a61ddbbec7a63923432da9
paper_authors.name: Zeynab Samei
paper_authors._id: 64a61ddbbec7a63923432daa
paper_authors.name: Shirin Dehghani
paper_authors._id: 64a61ddbbec7a63923432dab
paper_authors.name: Juan Diego Bermeo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-06T14:42:53.188Z
paper_authors._id: 64a6cbe6d09682887da5e818
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pdPBtj0fxrsucR5BbfNT2.jpeg
paper_authors.fullname: Juan Diego Bermeo
paper_authors.type: user
paper_authors.user: jdbermeo
paper_authors._id: 64a61ddbbec7a63923432dac
paper_authors.name: Maria Korobeynikova
paper_authors._id: 64a61ddbbec7a63923432dad
paper_authors.name: Fabrizio Gilardi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-06T14:43:01.395Z
paper_authors._id: 64a6ca8a9745644fc155cebd
paper_authors.avatarUrl: /avatars/cf0e998ea8d6ec1e32e7adc33feba2ed.svg
paper_authors.fullname: Fabrizio Gilardi
paper_authors.type: user
paper_authors.user: fgilardi
paper_title: Open-Source Large Language Models Outperform Crowd Workers and Approach
    ChatGPT in Text-Annotation Tasks
paper_summary: This study examines the performance of open-source Large Language Models
  (LLMs) in text annotation tasks and compares it with proprietary models like
  ChatGPT and human-based services such as MTurk. While prior research
  demonstrated the high performance of ChatGPT across numerous NLP tasks,
  open-source LLMs like HugginChat and FLAN are gaining attention for their
  cost-effectiveness, transparency, reproducibility, and superior data
  protection. We assess these models using both zero-shot and few-shot approaches
  and different temperature parameters across a range of text annotation tasks.
  Our findings show that while ChatGPT achieves the best performance in most
  tasks, open-source LLMs not only outperform MTurk but also demonstrate
  competitive potential against ChatGPT in specific tasks.
paper_ai_summary: Open-source LLMs demonstrate competitive performance in text annotation tasks against proprietary models and human-based services.
paper_ai_keywords: Large Language Models LLMs HugginChat FLAN zero-shot few-shot temperature parameters text annotation tasks