A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages
<|reference_start|>A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages: How should text dataset sizes be compared across languages? Even for content-matched (parallel) corpora, UTF-8 encoded text can require a dramatically different number of bytes for different languages. In our work, we define the byte premium between two languages as the ratio of bytes used to encode content-matched text in those languages. We compute byte premiums for 1155 languages, and we use linear regressions to estimate byte premiums for other languages. We release a tool to obtain byte premiums for any two languages, enabling comparisons of dataset sizes across languages for more equitable multilingual model development and data practices.<|reference_end|>
@article{arnett2024a,
    title={A Bit of a Problem: Measurement Disparities in Dataset Sizes Across
  Languages},
    author={Catherine Arnett, Tyler A. Chang, Benjamin K. Bergen},
    journal={arXiv preprint arXiv:2403.00686},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.00686},
    primaryClass={cs.CL}
}