Generating bilingual example sentences with large language models as lexicography assistants
<|reference_start|>Generating bilingual example sentences with large language models as lexicography assistants: We present a study of LLMs' performance in generating and rating example sentences for bilingual dictionaries across languages with varying resource levels: French (high-resource), Indonesian (mid-resource), and Tetun (low-resource), with English as the target language. We evaluate the quality of LLM-generated examples against the GDEX (Good Dictionary EXample) criteria: typicality, informativeness, and intelligibility. Our findings reveal that while LLMs can generate reasonably good dictionary examples, their performance degrades significantly for lower-resourced languages. We also observe high variability in human preferences for example quality, reflected in low inter-annotator agreement rates. To address this, we demonstrate that in-context learning can successfully align LLMs with individual annotator preferences. Additionally, we explore the use of pre-trained language models for automated rating of examples, finding that sentence perplexity serves as a good proxy for typicality and intelligibility in higher-resourced languages. Our study also contributes a novel dataset of 600 ratings for LLM-generated sentence pairs, and provides insights into the potential of LLMs in reducing the cost of lexicographic work, particularly for low-resource languages.<|reference_end|>
@article{merx2024generating,
    title={Generating bilingual example sentences with large language models as
  lexicography assistants},
    author={Raphael Merx, Ekaterina Vylomova, Kemal Kurniawan},
    journal={arXiv preprint arXiv:2410.03182},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03182},
    primaryClass={cs.CL}
}