title: Shiksha: A Technical Domain focused Translation Dataset and Model for
    Indian Languages
summary: Neural Machine Translation (NMT) models are typically trained on datasets
  with limited exposure to Scientific, Technical and Educational domains.
  Translation models thus, in general, struggle with tasks that involve
  scientific understanding or technical jargon. Their performance is found to be
  even worse for low-resource Indian languages. Finding a translation dataset
  that tends to these domains in particular, poses a difficult challenge. In this
  paper, we address this by creating a multilingual parallel corpus containing
  more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality
  translation pairs across 8 Indian languages. We achieve this by bitext mining
  human-translated transcriptions of NPTEL video lectures. We also finetune and
  evaluate NMT models using this corpus and surpass all other publicly available
  models at in-domain tasks. We also demonstrate the potential for generalizing
  to out-of-domain translation tasks by improving the baseline by over 2 BLEU on
  average for these Indian languages on the Flores+ benchmark. We are pleased to
  release our model and dataset via this link: https://huggingface.co/SPRINGLab.
paper_authors._id: 675baa5ccd0640cdf19e6784
paper_authors.name: Advait Joglekar
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-13T05:30:34.734Z
paper_authors._id: 62fa1718c95d426ff8eeec2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1718c95d426ff8eeec2a/pWb33DlH6NPtl9Avss0gA.png
paper_authors.fullname: Advait Joglekar
paper_authors.type: user
paper_authors.user: rumourscape
paper_authors._id: 675baa5ccd0640cdf19e6785
paper_authors.name: Srinivasan Umesh
paper_title: Shiksha: A Technical Domain focused Translation Dataset and Model for
    Indian Languages
paper_summary: Neural Machine Translation (NMT) models are typically trained on datasets
  with limited exposure to Scientific, Technical and Educational domains.
  Translation models thus, in general, struggle with tasks that involve
  scientific understanding or technical jargon. Their performance is found to be
  even worse for low-resource Indian languages. Finding a translation dataset
  that tends to these domains in particular, poses a difficult challenge. In this
  paper, we address this by creating a multilingual parallel corpus containing
  more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality
  translation pairs across 8 Indian languages. We achieve this by bitext mining
  human-translated transcriptions of NPTEL video lectures. We also finetune and
  evaluate NMT models using this corpus and surpass all other publicly available
  models at in-domain tasks. We also demonstrate the potential for generalizing
  to out-of-domain translation tasks by improving the baseline by over 2 BLEU on
  average for these Indian languages on the Flores+ benchmark. We are pleased to
  release our model and dataset via this link: https://huggingface.co/SPRINGLab.
paper_ai_summary: A new multilingual corpus from NPTEL video lectures and fine-tuned NMT models improve translation performance for scientific and technical domains, particularly low-resource Indian languages.
paper_ai_keywords: Neural Machine Translation NMT multilingual parallel corpus English-to-Indic Indic-to-Indic NPTEL bitext mining fine-tuning BLEU Flores+ benchmark