title: In-Context Example Selection via Similarity Search Improves Low-Resource
    Machine Translation
summary: The ability of generative large language models (LLMs) to perform in-context
  learning has given rise to a large body of research into how best to prompt
  models for various natural language processing tasks. In this paper, we focus
  on machine translation (MT), a task that has been shown to benefit from
  in-context translation examples. However no systematic studies have been
  published on how best to select examples, and mixed results have been reported
  on the usefulness of similarity-based selection over random selection. We
  provide a study covering multiple LLMs and multiple in-context example
  retrieval strategies, comparing multilingual sentence embeddings. We cover
  several language directions, representing different levels of language
  resourcedness (English into French, German, Swahili and Wolof). Contrarily to
  previously published results, we find that sentence embedding similarity can
  improve MT, especially for low-resource language directions, and discuss the
  balance between selection pool diversity and quality. We also highlight
  potential problems with the evaluation of LLM-based MT and suggest a more
  appropriate evaluation protocol, adapting the COMET metric to the evaluation of
  LLMs. Code and outputs are freely available at
  https://github.com/ArmelRandy/ICL-MT.
paper_authors._id: 66b08c099c752f3da4e64c84
paper_authors.name: Armel Zebaze
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:17:45.165Z
paper_authors._id: 63952b31ee411d4c6fba891c
paper_authors.avatarUrl: /avatars/e901c3aaa16d3a06dff09896ce8a67a2.svg
paper_authors.fullname: Armel Randy Zebaze
paper_authors.type: user
paper_authors.user: ArmelRandy
paper_authors._id: 66b08c099c752f3da4e64c85
paper_authors.name: Benoît Sagot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:17:52.065Z
paper_authors._id: 602ba2a739515f8d31237967
paper_authors.avatarUrl: /avatars/ef534712ca682bf74ec7eef17d3d1b5f.svg
paper_authors.fullname: Benoît Sagot
paper_authors.type: user
paper_authors.user: sagot
paper_authors._id: 66b08c099c752f3da4e64c86
paper_authors.name: Rachel Bawden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:17:57.687Z
paper_authors._id: 613a0c85e63aa832a98cecc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1653912785529-613a0c85e63aa832a98cecc4.png
paper_authors.fullname: Rachel Bawden
paper_authors.type: user
paper_authors.user: rbawden
paper_title: In-Context Example Selection via Similarity Search Improves Low-Resource
    Machine Translation
paper_summary: The ability of generative large language models (LLMs) to perform in-context
  learning has given rise to a large body of research into how best to prompt
  models for various natural language processing tasks. In this paper, we focus
  on machine translation (MT), a task that has been shown to benefit from
  in-context translation examples. However no systematic studies have been
  published on how best to select examples, and mixed results have been reported
  on the usefulness of similarity-based selection over random selection. We
  provide a study covering multiple LLMs and multiple in-context example
  retrieval strategies, comparing multilingual sentence embeddings. We cover
  several language directions, representing different levels of language
  resourcedness (English into French, German, Swahili and Wolof). Contrarily to
  previously published results, we find that sentence embedding similarity can
  improve MT, especially for low-resource language directions, and discuss the
  balance between selection pool diversity and quality. We also highlight
  potential problems with the evaluation of LLM-based MT and suggest a more
  appropriate evaluation protocol, adapting the COMET metric to the evaluation of
  LLMs. Code and outputs are freely available at
  https://github.com/ArmelRandy/ICL-MT.
paper_ai_summary: In-context translation using sentence embeddings improves machine translation performance, particularly for low-resource languages, by balancing selection pool diversity and quality.
paper_ai_keywords: generative large language models in-context learning machine translation multilingual sentence embeddings low-resource languages COMET metric LLM-based MT