Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.

PDF Abstract Findings of 2020 PDF Findings of 2020 Abstract

Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


  Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods
{"cc_re__cc_re__cc_by_sa_4_0": [" xmlns=\"http://www.w3.org/2000/svg\" width=\"512\" height=\"512\" viewBox=\"0 0 512 512\"><path d=\"M424 80H88a56.06 56.06 0 0 0-56 56v240a56.06 56.06 0 0 0 56 56h336a56.06 56.06 0 0 0 56-56V136a56.06 56.06 0 0 0-56-56zm-14.18 92.63l-144 112a16 16 0 0 1-19.64 0l-144-112a16 16 0 1 1 19.64-25.26L256 251.73l134.18-104.36a16 16 0 0 1 19.64 25.26z\"/></svg></span> hello@paperswithcode.com\n    </a>.\n    <span class=\"footer-contact-item\">\n        Papers With Code is a free resource with all data licensed under <a rel=\"noreferrer\" href=\"https://creativecommons.org/licenses/by-sa/4.0/\">CC-BY-SA</a>.\n    </span>\n  </div>\n\n  <div class=\"footer-links\">\n      <a href=\"/site/terms\">Terms</a>\n      <a href=\"/site/data-policy\">Data policy</a>\n      <a href=\"/site/cookies-policy\">Cookies policy</a>\n      <a href=\"/about#team\" class=\"fair-logo\"> from\n          <img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAANAAAAAgCAMAAABU6AZfAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAJcEhZcwAAFiUAABYlAUlSJPAAAABFUExURUdwTBwqMhwqMxsqMhkqMxsqMhwqMgCA+hwrMxJIgBsrMxsqMgJ28AF58wF38BsqMwB58hsqMwF17"]}