Open-Source Large Language Models Outperform Crowd Workers and Approach
  ChatGPT in Text-Annotation Tasks
This study examines the performance of open-source Large Language Models
(LLMs) in text annotation tasks and compares it with proprietary models like
ChatGPT and human-based services such as MTurk. While prior research
demonstrated the high performance of ChatGPT across numerous NLP tasks,
open-source LLMs like HugginChat and FLAN are gaining attention for their
cost-effectiveness, transparency, reproducibility, and superior data
protection. We assess these models using both zero-shot and few-shot approaches
and different temperature parameters across a range of text annotation tasks.
Our findings show that while ChatGPT achieves the best performance in most
tasks, open-source LLMs not only outperform MTurk but also demonstrate
competitive potential against ChatGPT in specific tasks.
64a61ddbbec7a63923432da7
Meysam Alizadeh
64a61ddbbec7a63923432da8
Maël Kubli
claimed_verified
2023-07-06T14:42:57.481Z
64a6cb90eb672ec8d53737c3
https://cdn-avatars.huggingface.co/v1/production/uploads/64a6cb90eb672ec8d53737c3/eEICiyB6q_Tv4BYTjOOx8.jpeg
Mael Kubli
user
kublima
64a61ddbbec7a63923432da9
Zeynab Samei
64a61ddbbec7a63923432daa
Shirin Dehghani
64a61ddbbec7a63923432dab
Juan Diego Bermeo
claimed_verified
2023-07-06T14:42:53.188Z
64a6cbe6d09682887da5e818
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pdPBtj0fxrsucR5BbfNT2.jpeg
Juan Diego Bermeo
user
jdbermeo
64a61ddbbec7a63923432dac
Maria Korobeynikova
64a61ddbbec7a63923432dad
Fabrizio Gilardi
claimed_verified
2023-07-06T14:43:01.395Z
64a6ca8a9745644fc155cebd
/avatars/cf0e998ea8d6ec1e32e7adc33feba2ed.svg
Fabrizio Gilardi
user
fgilardi
Open-Source Large Language Models Outperform Crowd Workers and Approach
  ChatGPT in Text-Annotation Tasks
This study examines the performance of open-source Large Language Models
(LLMs) in text annotation tasks and compares it with proprietary models like
ChatGPT and human-based services such as MTurk. While prior research
demonstrated the high performance of ChatGPT across numerous NLP tasks,
open-source LLMs like HugginChat and FLAN are gaining attention for their
cost-effectiveness, transparency, reproducibility, and superior data
protection. We assess these models using both zero-shot and few-shot approaches
and different temperature parameters across a range of text annotation tasks.
Our findings show that while ChatGPT achieves the best performance in most
tasks, open-source LLMs not only outperform MTurk but also demonstrate
competitive potential against ChatGPT in specific tasks.
Open-source LLMs demonstrate competitive performance in text annotation tasks against proprietary models and human-based services.
Large Language Models LLMs HugginChat FLAN zero-shot few-shot temperature parameters text annotation tasks