title: Aya Dataset: An Open-Access Collection for Multilingual Instruction
    Tuning
summary: Datasets are foundational to many breakthroughs in modern artificial
  intelligence. Many recent achievements in the space of natural language
  processing (NLP) can be attributed to the finetuning of pre-trained models on a
  diverse set of tasks that enables a large language model (LLM) to respond to
  instructions. Instruction fine-tuning (IFT) requires specifically constructed
  and annotated datasets. However, existing datasets are almost all in the
  English language. In this work, our primary goal is to bridge the language gap
  by building a human-curated instruction-following dataset spanning 65
  languages. We worked with fluent speakers of languages from around the world to
  collect natural instances of instructions and completions. Furthermore, we
  create the most extensive multilingual collection to date, comprising 513
  million instances through templating and translating existing datasets across
  114 languages. In total, we contribute four key resources: we develop and
  open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection,
  and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case
  study in participatory research, involving collaborators from 119 countries. We
  see this as a valuable framework for future research collaborations that aim to
  bridge gaps in resources.
paper_authors._id: 65c97c0e957afc29cea9febe
paper_authors.name: Shivalika Singh
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-06T11:02:40.683Z
paper_authors._id: 62c2175d756039dd0dd20509
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
paper_authors.fullname: Shivalika Singh
paper_authors.type: user
paper_authors.user: shivalikasingh
paper_authors._id: 65c97c0e957afc29cea9febf
paper_authors.name: Freddie Vargus
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T08:39:14.000Z
paper_authors._id: 618cb927ec9a3fd349fa7e26
paper_authors.avatarUrl: /avatars/75c81a484fcafac3b40ed6ca01832faf.svg
paper_authors.fullname: Freddie Vargus
paper_authors.type: user
paper_authors.user: freddie
paper_authors._id: 65c97c0e957afc29cea9fec0
paper_authors.name: Daniel Dsouza
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:21:25.120Z
paper_authors._id: 657495cbec3bf96e433489a2
paper_authors.avatarUrl: /avatars/3d0f05abaa8c87d37d06521a19d41aee.svg
paper_authors.fullname: Daniel Dsouza
paper_authors.type: user
paper_authors.user: DanielDsouza
paper_authors._id: 65c97c0e957afc29cea9fec1
paper_authors.name: Börje F. Karlsson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:21:34.285Z
paper_authors._id: 61e52be53d6dbb1da842316a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
paper_authors.fullname: Börje Karlsson
paper_authors.type: user
paper_authors.user: tellarin
paper_authors._id: 65c97c0e957afc29cea9fec2
paper_authors.name: Abinaya Mahendiran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:21:41.617Z
paper_authors._id: 5f3621f67e583543386216d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652247198455-5f3621f67e583543386216d1.jpeg
paper_authors.fullname: Abinaya Mahendiran
paper_authors.type: user
paper_authors.user: abinayam
paper_authors._id: 65c97c0e957afc29cea9fec3
paper_authors.name: Wei-Yin Ko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:21:49.033Z
paper_authors._id: 640d286c9e5247967ff69840
paper_authors.avatarUrl: /avatars/01b39240236e41859f7a9fd71f83382a.svg
paper_authors.fullname: Wei-Yin Ko
paper_authors.type: user
paper_authors.user: weiyinko
paper_authors._id: 65c97c0e957afc29cea9fec4
paper_authors.name: Herumb Shandilya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T13:21:24.380Z
paper_authors._id: 603944ad5590c916b80f84fc
paper_authors.avatarUrl: /avatars/265f11866d11666ae8e9145da8645ea7.svg
paper_authors.fullname: Herumb Shandilya
paper_authors.type: user
paper_authors.user: krypticmouse
paper_authors._id: 65c97c0e957afc29cea9fec5
paper_authors.name: Jay Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-14T08:43:00.444Z
paper_authors._id: 64b965493269cb9386c507c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b965493269cb9386c507c3/v_HEnaVVg9OzMZGDKfAk0.jpeg
paper_authors.fullname: Jay Patel
paper_authors.type: user
paper_authors.user: jpatel0057
paper_authors._id: 65c97c0e957afc29cea9fec6
paper_authors.name: Deividas Mataciunas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-13T15:33:33.851Z
paper_authors._id: 6040a00558b78f3a0047c23a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6040a00558b78f3a0047c23a/_BsyJoaCBO3r6GnfSgnIA.jpeg
paper_authors.fullname: David Mataciunas
paper_authors.type: user
paper_authors.user: DeividasM
paper_authors._id: 65c97c0e957afc29cea9fec7
paper_authors.name: Laura OMahony
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T13:21:25.987Z
paper_authors._id: 64b087aae55e77179863990d
paper_authors.avatarUrl: /avatars/214f61f74636fd69b58d534993ec79d7.svg
paper_authors.fullname: Laura O'Mahony
paper_authors.type: user
paper_authors.user: lomahony
paper_authors._id: 65c97c0e957afc29cea9fec8
paper_authors.name: Mike Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T08:39:09.444Z
paper_authors._id: 60d33fbbd7b174177faabd4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg
paper_authors.fullname: Mike Zhang
paper_authors.type: user
paper_authors.user: jjzha
paper_authors._id: 65c97c0e957afc29cea9fec9
paper_authors.name: Ramith Hettiarachchi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T16:14:31.431Z
paper_authors._id: 65765e39a90ae2daae06a4a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65765e39a90ae2daae06a4a6/cokha9RRWpvyvc4K8hkUI.jpeg
paper_authors.fullname: Ramith Hettiarachchi
paper_authors.type: user
paper_authors.user: ramithuh
paper_authors._id: 65c97c0e957afc29cea9feca
paper_authors.name: Joseph Wilson
paper_authors._id: 65c97c0e957afc29cea9fecb
paper_authors.name: Marina Machado
paper_authors._id: 65c97c0e957afc29cea9fecc
paper_authors.name: Luisa Souza Moura
paper_authors._id: 65c97c0e957afc29cea9fecd
paper_authors.name: Dominik Krzemiński
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T10:58:17.251Z
paper_authors._id: 662cc845616128914a3c9817
paper_authors.avatarUrl: /avatars/cd186de00f28bb866abc1ab6c4465663.svg
paper_authors.fullname: DomKrz
paper_authors.type: user
paper_authors.user: dokato
paper_authors._id: 65c97c0e957afc29cea9fece
paper_authors.name: Hakimeh Fadaei
paper_authors._id: 65c97c0e957afc29cea9fecf
paper_authors.name: Irem Ergün
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:24:08.102Z
paper_authors._id: 65c74cb0e2ba059be0024f6e
paper_authors.avatarUrl: /avatars/a052488893789bc47f773f866a300ed8.svg
paper_authors.fullname: Irem Ergun
paper_authors.type: user
paper_authors.user: irombie
paper_authors._id: 65c97c0e957afc29cea9fed0
paper_authors.name: Ifeoma Okoh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-13T07:50:01.621Z
paper_authors._id: 662eacf8b1346ef109a11445
paper_authors.avatarUrl: /avatars/44b4a3f4364eaa222a75fa1b5d1e48b5.svg
paper_authors.fullname: Ify Okoh
paper_authors.type: user
paper_authors.user: Ifyokoh
paper_authors._id: 65c97c0e957afc29cea9fed1
paper_authors.name: Aisha Alaagib
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-01T15:39:36.027Z
paper_authors._id: 66aba64690d50bf132d98477
paper_authors.avatarUrl: /avatars/61d0a6af2c3a7af5b0003bde3002b22d.svg
paper_authors.fullname: Aisha Alaagib Alryeh Mkean
paper_authors.type: user
paper_authors.user: aishaalaagib
paper_authors._id: 65c97c0e957afc29cea9fed2
paper_authors.name: Oshan Mudannayake
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:24:21.841Z
paper_authors._id: 62c92482d0b7de324a98a4de
paper_authors.avatarUrl: /avatars/0c4b25d0007439d715f44a239a8ea4d2.svg
paper_authors.fullname: Oshan Mudannayake
paper_authors.type: user
paper_authors.user: ivantha
paper_authors._id: 65c97c0e957afc29cea9fed3
paper_authors.name: Zaid Alyafeai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T08:39:11.757Z
paper_authors._id: 5f04bd384ec31d33a72116d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594145966049-noauth.jpeg
paper_authors.fullname: Zaid Alyafeai
paper_authors.type: user
paper_authors.user: Zaid
paper_authors._id: 65c97c0e957afc29cea9fed4
paper_authors.name: Vu Minh Chien
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-13T10:28:49.834Z
paper_authors._id: 60535c9d10aba34e3b6a2ef7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
paper_authors.fullname: vumichien
paper_authors.type: user
paper_authors.user: vumichien
paper_authors._id: 65c97c0e957afc29cea9fed5
paper_authors.name: Sebastian Ruder
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:24:36.688Z
paper_authors._id: 624575789dc26c59bae6a19e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648719221694-noauth.jpeg
paper_authors.fullname: Sebastian Ruder
paper_authors.type: user
paper_authors.user: ruder
paper_authors._id: 65c97c0e957afc29cea9fed6
paper_authors.name: Surya Guthikonda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-12T13:21:28.647Z
paper_authors._id: 644406a81bc692d87b263f8a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644406a81bc692d87b263f8a/Kj0YkRnrei1jSynSP2_Mx.jpeg
paper_authors.fullname: Surya Guthikonda
paper_authors.type: user
paper_authors.user: SuryaKrishna02
paper_authors._id: 65c97c0e957afc29cea9fed7
paper_authors.name: Emad A. Alghamdi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:24:47.118Z
paper_authors._id: 62b02bfe744b9a896b9993d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b02bfe744b9a896b9993d3/ZtNOB3S1a9yQCouMwteAW.jpeg
paper_authors.fullname: Emad Alghamdi
paper_authors.type: user
paper_authors.user: eaalghamdi
paper_authors._id: 65c97c0e957afc29cea9fed8
paper_authors.name: Sebastian Gehrmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:24:56.163Z
paper_authors._id: 618a8d67cabd3c4c8e4481f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636661214776-618a8d67cabd3c4c8e4481f0.jpeg
paper_authors.fullname: Sebastian Gehrmann
paper_authors.type: user
paper_authors.user: SebGehr
paper_authors._id: 65c97c0e957afc29cea9fed9
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:25:03.746Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 65c97c0e957afc29cea9feda
paper_authors.name: Max Bartolo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:25:10.733Z
paper_authors._id: 604f8c093050a33ebb17ef66
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615825896626-noauth.jpeg
paper_authors.fullname: Max Bartolo
paper_authors.type: user
paper_authors.user: mbartolo
paper_authors._id: 65c97c0e957afc29cea9fedb
paper_authors.name: Julia Kreutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:25:31.374Z
paper_authors._id: 6544e43b12da508864c38f96
paper_authors.avatarUrl: /avatars/76f0cd55b4bf9c03d2686e146c6f795f.svg
paper_authors.fullname: Julia Kreutzer
paper_authors.type: user
paper_authors.user: JuliaKreutzerCohere
paper_authors._id: 65c97c0e957afc29cea9fedc
paper_authors.name: Ahmet Üstün
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:25:40.837Z
paper_authors._id: 60d35d7ad7b174177faabd5b
paper_authors.avatarUrl: /avatars/4e5403b9d4a845a2d21e8217dc3c16d2.svg
paper_authors.fullname: Ahmet Üstün
paper_authors.type: user
paper_authors.user: ahmetu
paper_authors._id: 65c97c0e957afc29cea9fedd
paper_authors.name: Marzieh Fadaee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-21T12:06:18.497Z
paper_authors._id: 6441042d5d600fb0951a5f99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
paper_authors.fullname: Marzieh Fadaee
paper_authors.type: user
paper_authors.user: MarziehFadaee
paper_authors._id: 65c97c0e957afc29cea9fede
paper_authors.name: Sara Hooker
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-27T03:43:42.605Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_title: Aya Dataset: An Open-Access Collection for Multilingual Instruction
    Tuning
paper_summary: Datasets are foundational to many breakthroughs in modern artificial
  intelligence. Many recent achievements in the space of natural language
  processing (NLP) can be attributed to the finetuning of pre-trained models on a
  diverse set of tasks that enables a large language model (LLM) to respond to
  instructions. Instruction fine-tuning (IFT) requires specifically constructed
  and annotated datasets. However, existing datasets are almost all in the
  English language. In this work, our primary goal is to bridge the language gap
  by building a human-curated instruction-following dataset spanning 65
  languages. We worked with fluent speakers of languages from around the world to
  collect natural instances of instructions and completions. Furthermore, we
  create the most extensive multilingual collection to date, comprising 513
  million instances through templating and translating existing datasets across
  114 languages. In total, we contribute four key resources: we develop and
  open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection,
  and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case
  study in participatory research, involving collaborators from 119 countries. We
  see this as a valuable framework for future research collaborations that aim to
  bridge gaps in resources.
paper_ai_summary: The initiative builds a human-curated instruction-following dataset spanning 65 languages and creates the largest multilingual collection of instruction-following instances through templating and translating existing datasets across 114 languages, contributing datasets and platforms for participatory research.
paper_ai_keywords: Instruction fine-tuning IFT large language model LLm Aya Annotation Platform Aya Dataset Aya Collection Aya Evaluation Suite participatory research