title: Learning Language-Specific Layers for Multilingual Machine Translation
summary: Multilingual Machine Translation promises to improve translation quality
  between non-English languages. This is advantageous for several reasons, namely
  lower latency (no need to translate twice), and reduced error cascades (e.g.,
  avoiding losing gender and formality information when translating through
  English). On the downside, adding more languages reduces model capacity per
  language, which is usually countered by increasing the overall model size,
  making training harder and inference slower. In this work, we introduce
  Language-Specific Transformer Layers (LSLs), which allow us to increase model
  capacity, while keeping the amount of computation and the number of parameters
  used in the forward pass constant. The key idea is to have some layers of the
  encoder be source or target language-specific, while keeping the remaining
  layers shared. We study the best way to place these layers using a neural
  architecture search inspired approach, and achieve an improvement of 1.3 chrF
  (1.5 spBLEU) points over not using LSLs on a separate decoder architecture, and
  1.9 chrF (2.2 spBLEU) on a shared decoder one.
paper_authors._id: 6454791ccd09ceba0e169e88
paper_authors.name: Telmo Pessoa Pires
paper_authors._id: 6454791ccd09ceba0e169e89
paper_authors.name: Robin M. Schmidt
paper_authors._id: 6454791ccd09ceba0e169e8a
paper_authors.name: Yi-Hsiu Liao
paper_authors._id: 6454791ccd09ceba0e169e8b
paper_authors.name: Stephan Peitz
paper_title: Learning Language-Specific Layers for Multilingual Machine Translation
paper_summary: Multilingual Machine Translation promises to improve translation quality
  between non-English languages. This is advantageous for several reasons, namely
  lower latency (no need to translate twice), and reduced error cascades (e.g.,
  avoiding losing gender and formality information when translating through
  English). On the downside, adding more languages reduces model capacity per
  language, which is usually countered by increasing the overall model size,
  making training harder and inference slower. In this work, we introduce
  Language-Specific Transformer Layers (LSLs), which allow us to increase model
  capacity, while keeping the amount of computation and the number of parameters
  used in the forward pass constant. The key idea is to have some layers of the
  encoder be source or target language-specific, while keeping the remaining
  layers shared. We study the best way to place these layers using a neural
  architecture search inspired approach, and achieve an improvement of 1.3 chrF
  (1.5 spBLEU) points over not using LSLs on a separate decoder architecture, and
  1.9 chrF (2.2 spBLEU) on a shared decoder one.
paper_ai_summary: Language-Specific Transformer Layers enhance multilingual machine translation quality without increasing computational cost or parameters.
paper_ai_keywords: Language-Specific Transformer Layers Transformer Layers neural architecture search chrF spBLEU