XferBench: a Data-Driven Benchmark for Emergent Language
<|reference_start|>XferBench: a Data-Driven Benchmark for Emergent Language: In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines.<|reference_end|>
@article{boldt2024xferbench:,
    title={XferBench: a Data-Driven Benchmark for Emergent Language},
    author={Brendon Boldt, David Mortensen},
    journal={Proceedings of the 2024 Conference of the North American Chapter
  of the Association for Computational Linguistics: Human Language Technologies
  (Volume 1: Long Papers), 1475-1489},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.03456},
    primaryClass={cs.CL}
}