In-Context Example Selection via Similarity Search Improves Low-Resource
  Machine Translation
The ability of generative large language models (LLMs) to perform in-context
learning has given rise to a large body of research into how best to prompt
models for various natural language processing tasks. In this paper, we focus
on machine translation (MT), a task that has been shown to benefit from
in-context translation examples. However no systematic studies have been
published on how best to select examples, and mixed results have been reported
on the usefulness of similarity-based selection over random selection. We
provide a study covering multiple LLMs and multiple in-context example
retrieval strategies, comparing multilingual sentence embeddings. We cover
several language directions, representing different levels of language
resourcedness (English into French, German, Swahili and Wolof). Contrarily to
previously published results, we find that sentence embedding similarity can
improve MT, especially for low-resource language directions, and discuss the
balance between selection pool diversity and quality. We also highlight
potential problems with the evaluation of LLM-based MT and suggest a more
appropriate evaluation protocol, adapting the COMET metric to the evaluation of
LLMs. Code and outputs are freely available at
https://github.com/ArmelRandy/ICL-MT.
66b08c099c752f3da4e64c84
Armel Zebaze
admin_assigned
2024-08-05T09:17:45.165Z
63952b31ee411d4c6fba891c
/avatars/e901c3aaa16d3a06dff09896ce8a67a2.svg
Armel Randy Zebaze
user
ArmelRandy
66b08c099c752f3da4e64c85
Benoît Sagot
admin_assigned
2024-08-05T09:17:52.065Z
602ba2a739515f8d31237967
/avatars/ef534712ca682bf74ec7eef17d3d1b5f.svg
Benoît Sagot
user
sagot
66b08c099c752f3da4e64c86
Rachel Bawden
admin_assigned
2024-08-05T09:17:57.687Z
613a0c85e63aa832a98cecc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1653912785529-613a0c85e63aa832a98cecc4.png
Rachel Bawden
user
rbawden
In-Context Example Selection via Similarity Search Improves Low-Resource
  Machine Translation
The ability of generative large language models (LLMs) to perform in-context
learning has given rise to a large body of research into how best to prompt
models for various natural language processing tasks. In this paper, we focus
on machine translation (MT), a task that has been shown to benefit from
in-context translation examples. However no systematic studies have been
published on how best to select examples, and mixed results have been reported
on the usefulness of similarity-based selection over random selection. We
provide a study covering multiple LLMs and multiple in-context example
retrieval strategies, comparing multilingual sentence embeddings. We cover
several language directions, representing different levels of language
resourcedness (English into French, German, Swahili and Wolof). Contrarily to
previously published results, we find that sentence embedding similarity can
improve MT, especially for low-resource language directions, and discuss the
balance between selection pool diversity and quality. We also highlight
potential problems with the evaluation of LLM-based MT and suggest a more
appropriate evaluation protocol, adapting the COMET metric to the evaluation of
LLMs. Code and outputs are freely available at
https://github.com/ArmelRandy/ICL-MT.
In-context translation using sentence embeddings improves machine translation performance, particularly for low-resource languages, by balancing selection pool diversity and quality.
generative large language models in-context learning machine translation multilingual sentence embeddings low-resource languages COMET metric LLM-based MT