Aya Dataset: An Open-Access Collection for Multilingual Instruction
  Tuning
Datasets are foundational to many breakthroughs in modern artificial
intelligence. Many recent achievements in the space of natural language
processing (NLP) can be attributed to the finetuning of pre-trained models on a
diverse set of tasks that enables a large language model (LLM) to respond to
instructions. Instruction fine-tuning (IFT) requires specifically constructed
and annotated datasets. However, existing datasets are almost all in the
English language. In this work, our primary goal is to bridge the language gap
by building a human-curated instruction-following dataset spanning 65
languages. We worked with fluent speakers of languages from around the world to
collect natural instances of instructions and completions. Furthermore, we
create the most extensive multilingual collection to date, comprising 513
million instances through templating and translating existing datasets across
114 languages. In total, we contribute four key resources: we develop and
open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection,
and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case
study in participatory research, involving collaborators from 119 countries. We
see this as a valuable framework for future research collaborations that aim to
bridge gaps in resources.
65c97c0e957afc29cea9febe
Shivalika Singh
extracted_confirmed
2024-12-06T11:02:40.683Z
62c2175d756039dd0dd20509
https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
Shivalika Singh
user
shivalikasingh
65c97c0e957afc29cea9febf
Freddie Vargus
claimed_verified
2024-02-12T08:39:14.000Z
618cb927ec9a3fd349fa7e26
/avatars/75c81a484fcafac3b40ed6ca01832faf.svg
Freddie Vargus
user
freddie
65c97c0e957afc29cea9fec0
Daniel Dsouza
admin_assigned
2024-02-12T16:21:25.120Z
657495cbec3bf96e433489a2
/avatars/3d0f05abaa8c87d37d06521a19d41aee.svg
Daniel Dsouza
user
DanielDsouza
65c97c0e957afc29cea9fec1
Börje F. Karlsson
admin_assigned
2024-02-12T16:21:34.285Z
61e52be53d6dbb1da842316a
https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
Börje Karlsson
user
tellarin
65c97c0e957afc29cea9fec2
Abinaya Mahendiran
admin_assigned
2024-02-12T16:21:41.617Z
5f3621f67e583543386216d1
https://cdn-avatars.huggingface.co/v1/production/uploads/1652247198455-5f3621f67e583543386216d1.jpeg
Abinaya Mahendiran
user
abinayam
65c97c0e957afc29cea9fec3
Wei-Yin Ko
admin_assigned
2024-02-12T16:21:49.033Z
640d286c9e5247967ff69840
/avatars/01b39240236e41859f7a9fd71f83382a.svg
Wei-Yin Ko
user
weiyinko
65c97c0e957afc29cea9fec4
Herumb Shandilya
claimed_verified
2024-02-12T13:21:24.380Z
603944ad5590c916b80f84fc
/avatars/265f11866d11666ae8e9145da8645ea7.svg
Herumb Shandilya
user
krypticmouse
65c97c0e957afc29cea9fec5
Jay Patel
claimed_verified
2024-02-14T08:43:00.444Z
64b965493269cb9386c507c3
https://cdn-avatars.huggingface.co/v1/production/uploads/64b965493269cb9386c507c3/v_HEnaVVg9OzMZGDKfAk0.jpeg
Jay Patel
user
jpatel0057
65c97c0e957afc29cea9fec6
Deividas Mataciunas
claimed_verified
2024-02-13T15:33:33.851Z
6040a00558b78f3a0047c23a
https://cdn-avatars.huggingface.co/v1/production/uploads/6040a00558b78f3a0047c23a/_BsyJoaCBO3r6GnfSgnIA.jpeg
David Mataciunas
user
DeividasM
65c97c0e957afc29cea9fec7
Laura OMahony
claimed_verified
2024-02-12T13:21:25.987Z
64b087aae55e77179863990d
/avatars/214f61f74636fd69b58d534993ec79d7.svg
Laura O'Mahony
user
lomahony
65c97c0e957afc29cea9fec8
Mike Zhang
claimed_verified
2024-02-12T08:39:09.444Z
60d33fbbd7b174177faabd4f
https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg
Mike Zhang
user
jjzha
65c97c0e957afc29cea9fec9
Ramith Hettiarachchi
claimed_verified
2024-02-12T16:14:31.431Z
65765e39a90ae2daae06a4a6
https://cdn-avatars.huggingface.co/v1/production/uploads/65765e39a90ae2daae06a4a6/cokha9RRWpvyvc4K8hkUI.jpeg
Ramith Hettiarachchi
user
ramithuh
65c97c0e957afc29cea9feca
Joseph Wilson
65c97c0e957afc29cea9fecb
Marina Machado
65c97c0e957afc29cea9fecc
Luisa Souza Moura
65c97c0e957afc29cea9fecd
Dominik Krzemiński
claimed_verified
2025-02-24T10:58:17.251Z
662cc845616128914a3c9817
/avatars/cd186de00f28bb866abc1ab6c4465663.svg
DomKrz
user
dokato
65c97c0e957afc29cea9fece
Hakimeh Fadaei
65c97c0e957afc29cea9fecf
Irem Ergün
admin_assigned
2024-02-12T16:24:08.102Z
65c74cb0e2ba059be0024f6e
/avatars/a052488893789bc47f773f866a300ed8.svg
Irem Ergun
user
irombie
65c97c0e957afc29cea9fed0
Ifeoma Okoh
claimed_verified
2024-05-13T07:50:01.621Z
662eacf8b1346ef109a11445
/avatars/44b4a3f4364eaa222a75fa1b5d1e48b5.svg
Ify Okoh
user
Ifyokoh
65c97c0e957afc29cea9fed1
Aisha Alaagib
claimed_verified
2024-08-01T15:39:36.027Z
66aba64690d50bf132d98477
/avatars/61d0a6af2c3a7af5b0003bde3002b22d.svg
Aisha Alaagib Alryeh Mkean
user
aishaalaagib
65c97c0e957afc29cea9fed2
Oshan Mudannayake
admin_assigned
2024-02-12T16:24:21.841Z
62c92482d0b7de324a98a4de
/avatars/0c4b25d0007439d715f44a239a8ea4d2.svg
Oshan Mudannayake
user
ivantha
65c97c0e957afc29cea9fed3
Zaid Alyafeai
claimed_verified
2024-02-12T08:39:11.757Z
5f04bd384ec31d33a72116d1
https://cdn-avatars.huggingface.co/v1/production/uploads/1594145966049-noauth.jpeg
Zaid Alyafeai
user
Zaid
65c97c0e957afc29cea9fed4
Vu Minh Chien
claimed_verified
2024-02-13T10:28:49.834Z
60535c9d10aba34e3b6a2ef7
https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
vumichien
user
vumichien
65c97c0e957afc29cea9fed5
Sebastian Ruder
admin_assigned
2024-02-12T16:24:36.688Z
624575789dc26c59bae6a19e
https://cdn-avatars.huggingface.co/v1/production/uploads/1648719221694-noauth.jpeg
Sebastian Ruder
user
ruder
65c97c0e957afc29cea9fed6
Surya Guthikonda
claimed_verified
2024-02-12T13:21:28.647Z
644406a81bc692d87b263f8a
https://cdn-avatars.huggingface.co/v1/production/uploads/644406a81bc692d87b263f8a/Kj0YkRnrei1jSynSP2_Mx.jpeg
Surya Guthikonda
user
SuryaKrishna02
65c97c0e957afc29cea9fed7
Emad A. Alghamdi
admin_assigned
2024-02-12T16:24:47.118Z
62b02bfe744b9a896b9993d3
https://cdn-avatars.huggingface.co/v1/production/uploads/62b02bfe744b9a896b9993d3/ZtNOB3S1a9yQCouMwteAW.jpeg
Emad Alghamdi
user
eaalghamdi
65c97c0e957afc29cea9fed8
Sebastian Gehrmann
admin_assigned
2024-02-12T16:24:56.163Z
618a8d67cabd3c4c8e4481f0
https://cdn-avatars.huggingface.co/v1/production/uploads/1636661214776-618a8d67cabd3c4c8e4481f0.jpeg
Sebastian Gehrmann
user
SebGehr
65c97c0e957afc29cea9fed9
Niklas Muennighoff
admin_assigned
2024-02-12T16:25:03.746Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
65c97c0e957afc29cea9feda
Max Bartolo
admin_assigned
2024-02-12T16:25:10.733Z
604f8c093050a33ebb17ef66
https://cdn-avatars.huggingface.co/v1/production/uploads/1615825896626-noauth.jpeg
Max Bartolo
user
mbartolo
65c97c0e957afc29cea9fedb
Julia Kreutzer
admin_assigned
2024-02-12T16:25:31.374Z
6544e43b12da508864c38f96
/avatars/76f0cd55b4bf9c03d2686e146c6f795f.svg
Julia Kreutzer
user
JuliaKreutzerCohere
65c97c0e957afc29cea9fedc
Ahmet Üstün
admin_assigned
2024-02-12T16:25:40.837Z
60d35d7ad7b174177faabd5b
/avatars/4e5403b9d4a845a2d21e8217dc3c16d2.svg
Ahmet Üstün
user
ahmetu
65c97c0e957afc29cea9fedd
Marzieh Fadaee
extracted_confirmed
2024-03-21T12:06:18.497Z
6441042d5d600fb0951a5f99
https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
Marzieh Fadaee
user
MarziehFadaee
65c97c0e957afc29cea9fede
Sara Hooker
extracted_confirmed
2024-05-27T03:43:42.605Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
Aya Dataset: An Open-Access Collection for Multilingual Instruction
  Tuning
Datasets are foundational to many breakthroughs in modern artificial
intelligence. Many recent achievements in the space of natural language
processing (NLP) can be attributed to the finetuning of pre-trained models on a
diverse set of tasks that enables a large language model (LLM) to respond to
instructions. Instruction fine-tuning (IFT) requires specifically constructed
and annotated datasets. However, existing datasets are almost all in the
English language. In this work, our primary goal is to bridge the language gap
by building a human-curated instruction-following dataset spanning 65
languages. We worked with fluent speakers of languages from around the world to
collect natural instances of instructions and completions. Furthermore, we
create the most extensive multilingual collection to date, comprising 513
million instances through templating and translating existing datasets across
114 languages. In total, we contribute four key resources: we develop and
open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection,
and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case
study in participatory research, involving collaborators from 119 countries. We
see this as a valuable framework for future research collaborations that aim to
bridge gaps in resources.
The initiative builds a human-curated instruction-following dataset spanning 65 languages and creates the largest multilingual collection of instruction-following instances through templating and translating existing datasets across 114 languages, contributing datasets and platforms for participatory research.
Instruction fine-tuning IFT large language model LLm Aya Annotation Platform Aya Dataset Aya Collection Aya Evaluation Suite participatory research