Shiksha: A Technical Domain focused Translation Dataset and Model for
  Indian Languages
Neural Machine Translation (NMT) models are typically trained on datasets
with limited exposure to Scientific, Technical and Educational domains.
Translation models thus, in general, struggle with tasks that involve
scientific understanding or technical jargon. Their performance is found to be
even worse for low-resource Indian languages. Finding a translation dataset
that tends to these domains in particular, poses a difficult challenge. In this
paper, we address this by creating a multilingual parallel corpus containing
more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality
translation pairs across 8 Indian languages. We achieve this by bitext mining
human-translated transcriptions of NPTEL video lectures. We also finetune and
evaluate NMT models using this corpus and surpass all other publicly available
models at in-domain tasks. We also demonstrate the potential for generalizing
to out-of-domain translation tasks by improving the baseline by over 2 BLEU on
average for these Indian languages on the Flores+ benchmark. We are pleased to
release our model and dataset via this link: https://huggingface.co/SPRINGLab.
675baa5ccd0640cdf19e6784
Advait Joglekar
extracted_confirmed
2024-12-13T05:30:34.734Z
62fa1718c95d426ff8eeec2a
https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1718c95d426ff8eeec2a/pWb33DlH6NPtl9Avss0gA.png
Advait Joglekar
user
rumourscape
675baa5ccd0640cdf19e6785
Srinivasan Umesh
Shiksha: A Technical Domain focused Translation Dataset and Model for
  Indian Languages
Neural Machine Translation (NMT) models are typically trained on datasets
with limited exposure to Scientific, Technical and Educational domains.
Translation models thus, in general, struggle with tasks that involve
scientific understanding or technical jargon. Their performance is found to be
even worse for low-resource Indian languages. Finding a translation dataset
that tends to these domains in particular, poses a difficult challenge. In this
paper, we address this by creating a multilingual parallel corpus containing
more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality
translation pairs across 8 Indian languages. We achieve this by bitext mining
human-translated transcriptions of NPTEL video lectures. We also finetune and
evaluate NMT models using this corpus and surpass all other publicly available
models at in-domain tasks. We also demonstrate the potential for generalizing
to out-of-domain translation tasks by improving the baseline by over 2 BLEU on
average for these Indian languages on the Flores+ benchmark. We are pleased to
release our model and dataset via this link: https://huggingface.co/SPRINGLab.
A new multilingual corpus from NPTEL video lectures and fine-tuned NMT models improve translation performance for scientific and technical domains, particularly low-resource Indian languages.
Neural Machine Translation NMT multilingual parallel corpus English-to-Indic Indic-to-Indic NPTEL bitext mining fine-tuning BLEU Flores+ benchmark