readme: ---
  language: es
  license: cc-by-4.0
  tags:
  - spanish
  - roberta
  pipeline_tag: fill-mask
  datasets:
  - bertin-project/mc4-es-sampled
  widget:
  - text: Fui a la librería a comprar un <mask>.
  ---
  
  - [Version v2](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v2) (default): April 28th, 2022
  - [Version v1](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1): July 26th, 2021
  - [Version v1-512](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1-512): July 26th, 2021
  - [Version beta](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta): July 15th, 2021
  
  # BERTIN
  
  <div align=center>
  <img alt="BERTIN logo" src="https://huggingface.co/bertin-project/bertin-roberta-base-spanish/resolve/main/images/bertin.png" width="200px">
  </div>
  
  BERTIN is a series of BERT-based models for Spanish. The current model hub points to the best of all RoBERTa-base models trained from scratch on the Spanish portion of mC4 using [Flax](https://github.com/google/flax). All code and scripts are included.
  
  This is part of the
  [Flax/Jax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google Cloud.
  
  The aim of this project was to pre-train a RoBERTa-base model from scratch during the Flax/JAX Community Event, in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.
  
  
  ## Team members
  
  - Javier de la Rosa ([versae](https://huggingface.co/versae))
  - Eduardo González ([edugp](https://huggingface.co/edugp))
  - Paulo Villegas ([paulo](https://huggingface.co/paulo))
  - Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
  - Manu Romero ([mrm8488](https://huggingface.co/))
  - María Grandury ([mariagrandury](https://huggingface.co/))
  
  
  ## Citation and Related Information
  
  To cite this model:
  ```bibtex
  @article{BERTIN,
  	author = {Javier De la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury},
  	title = {BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling},
  	journal = {Procesamiento del Lenguaje Natural},
  	volume = {68},
  	number = {0},
  	year = {2022},
  	keywords = {},
  	abstract = {The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pretraining sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name perplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget.},
  	issn = {1989-7553},
  	url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6403},
  	pages = {13--23}
  }
  ```
  
  If you use this model, we would love to hear about it! Reach out on twitter, GitHub, Discord, or shoot us an email.
  
  ## Team
  
  - Javier de la Rosa ([versae](https://huggingface.co/versae))
  - Eduardo González ([edugp](https://huggingface.co/edugp))
  - Paulo Villegas ([paulo](https://huggingface.co/paulo))
  - Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
  - Manu Romero ([mrm8488](https://huggingface.co/))
  - María Grandury ([mariagrandury](https://huggingface.co/))
  
  ## Acknowledgements
  
  This project would not have been possible without compute generously provided by the Huggingface and Google through the
  [TPU Research Cloud](https://sites.research.google/trc/), as well as the Cloud TPU team for providing early access to the [Cloud TPU VM](https://cloud.google.com/blog/products/compute/introducing-cloud-tpu-vms).
  
  ## Disclaimer
  
  The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions. When third parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence. In no event shall the owner of the models be liable for any results arising from the use made by third parties of these models.
  
  <hr>
  
  
  <details>
  <summary>Full report</summary>
  
  # Motivation
  
  According to [Wikipedia](https://en.wikipedia.org/wiki/List_of_languages_by_total_number_