Dynaword: From One-shot to Continuously Developed Datasets
Large-scale datasets are foundational for research and development in natural
language processing. However, current approaches face three key challenges: (1)
reliance on ambiguously licensed sources restricting use, sharing, and
derivative works; (2) static dataset releases that prevent community
contributions and diminish longevity; and (3) quality assurance processes
restricted to publishing teams rather than leveraging community expertise.
  To address these limitations, we introduce two contributions: the Dynaword
approach and Danish Dynaword. The Dynaword approach is a framework for creating
large-scale, open datasets that can be continuously updated through community
collaboration. Danish Dynaword is a concrete implementation that validates this
approach and demonstrates its potential. Danish Dynaword contains over four
times as many tokens as comparable releases, is exclusively openly licensed,
and has received multiple contributions across industry and research. The
repository includes light-weight tests to ensure data formatting, quality, and
documentation, establishing a sustainable framework for ongoing community
contributions and dataset evolution.
6891f8409373cd501b6a2d82
Kenneth Enevoldsen
6891f8409373cd501b6a2d83
Kristian Nørgaard Jensen
6891f8409373cd501b6a2d84
Jan Kostkan
6891f8409373cd501b6a2d85
Balázs Szabó
6891f8409373cd501b6a2d86
Márton Kardos
6891f8409373cd501b6a2d87
Kirten Vad
6891f8409373cd501b6a2d88
Andrea Blasi Núñez
6891f8409373cd501b6a2d89
Gianluca Barmina
6891f8409373cd501b6a2d8a
Jacob Nielsen
6891f8409373cd501b6a2d8b
Rasmus Larsen
6891f8409373cd501b6a2d8c
Peter Vahlstrup
6891f8409373cd501b6a2d8d
Per Møldrup Dalum
6891f8409373cd501b6a2d8e
Desmond Elliott
6891f8409373cd501b6a2d8f
Lukas Galke
6891f8409373cd501b6a2d90
Peter Schneider-Kamp
6891f8409373cd501b6a2d91
Kristoffer Nielbo
Dynaword: From One-shot to Continuously Developed Datasets
Large-scale datasets are foundational for research and development in natural
language processing. However, current approaches face three key challenges: (1)
reliance on ambiguously licensed sources restricting use, sharing, and
derivative works; (2) static dataset releases that prevent community
contributions and diminish longevity; and (3) quality assurance processes
restricted to publishing teams rather than leveraging community expertise.
  To address these limitations, we introduce two contributions: the Dynaword
approach and Danish Dynaword. The Dynaword approach is a framework for creating
large-scale, open datasets that can be continuously updated through community
collaboration. Danish Dynaword is a concrete implementation that validates this
approach and demonstrates its potential. Danish Dynaword contains over four
times as many tokens as comparable releases, is exclusively openly licensed,
and has received multiple contributions across industry and research. The
repository includes light-weight tests to ensure data formatting, quality, and
documentation, establishing a sustainable framework for ongoing community
contributions and dataset evolution.
A framework called Dynaword and its implementation Danish Dynaword enable community-driven, open, and continuously updated large-scale natural language datasets.
Dynaword Danish Dynaword