readme: ---
  language: 
    - es
  tags:
  - biomedical
  - spanish
  license: apache-2.0
  metrics:
  - ppl
  widget:
  - text: "El único antecedente personal a reseñar era la <mask> arterial."
  - text: "Las radiologías óseas de cuerpo entero no detectan alteraciones <mask>, ni alteraciones vertebrales."
  - text: "En el <mask> toraco-abdómino-pélvico no se encontraron hallazgos patológicos de interés."
  ---
  
  # Biomedical language model for Spanish
  
  ## Table of contents
  <details>
  <summary>Click to expand</summary>
  
  - [Model description](#model-description)
  - [Intended uses and limitations](#intended-use)
  - [How to use](#how-to-use)
  - [Limitations and bias](#limitations-and-bias)
  - [Training](#training)
    - [Tokenization and model pretraining](#Tokenization-pretraining)
    - [Training corpora and preprocessing](#training-corpora-preprocessing)
  - [Evaluation](#evaluation)
  - [Additional information](#additional-information)
    - [Author](#author)
    - [Contact information](#contact-information)
    - [Copyright](#copyright)
    - [Licensing information](#licensing-information)
    - [Funding](#funding)
    - [Disclaimer](#disclaimer)
    
  </details>
  
  ## Model description
  Biomedical pretrained language model for Spanish. For more details about the corpus, the pretraining and the evaluation, check the official [repository](https://github.com/PlanTL-SANIDAD/lm-biomedical-clinical-es) and read our [preprint](https://arxiv.org/abs/2109.03570).
  
  ## Intended uses and limitations
  The model is ready-to-use only for masked language modelling to perform the Fill Mask task (try the inference API or read the next section). However, it is intended to be fine-tuned on downstream tasks such as Named Entity Recognition or Text Classification.
  
  
  ## How to use
  
  ```python
  from transformers import AutoTokenizer, AutoModelForMaskedLM
  tokenizer = AutoTokenizer.from_pretrained("BSC-TeMU/roberta-base-biomedical-es")
  model = AutoModelForMaskedLM.from_pretrained("BSC-TeMU/roberta-base-biomedical-es")
  from transformers import pipeline
  unmasker = pipeline('fill-mask', model="BSC-TeMU/roberta-base-biomedical-es")
  unmasker("El único antecedente personal a reseñar era la <mask> arterial.")
  ```
  ```
  # Output
  [
    {
      "sequence": " El único antecedente personal a reseñar era la hipertensión arterial.",
      "score": 0.9855039715766907,
      "token": 3529,
      "token_str": " hipertensión"
    },
    {
      "sequence": " El único antecedente personal a reseñar era la diabetes arterial.",
      "score": 0.0039140828885138035,
      "token": 1945,
      "token_str": " diabetes"
    },
    {
      "sequence": " El único antecedente personal a reseñar era la hipotensión arterial.",
      "score": 0.002484665485098958,
      "token": 11483,
      "token_str": " hipotensión"
    },
    {
      "sequence": " El único antecedente personal a reseñar era la Hipertensión arterial.",
      "score": 0.0023484621196985245,
      "token": 12238,
      "token_str": " Hipertensión"
    },
    {
      "sequence": " El único antecedente personal a reseñar era la presión arterial.",
      "score": 0.0008009297889657319,
      "token": 2267,
      "token_str": " presión"
    }
  ]
  ```
  
  ## Training
  
  ### Tokenization and model pretraining
  
  This model is a [RoBERTa-based](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model trained on a
  **biomedical** corpus in Spanish collected from several sources (see next section). 
  
  The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2)
  used in the original [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model with a vocabulary size of 52,000 tokens. The pretraining consists of a masked language model training at the subword level following the approach employed for the RoBERTa base model with the same hyperparameters as in the original work. The training lasted a total of 48 hours with 16 NVIDIA V100 GPUs of 16GB DDRAM, using Adam optimizer with a peak learning rate of 0.0005 and an effective batch size of 2,048 sentences.
  
  
  ### Training corpora and preprocessing
  
  The training corpus is composed of several biomedical corpora in Spanish, collected from publicly available corpora and crawlers.
  To obtain a high-quality training corpus, a cleaning pipeline with the following operations has been applied:
  
  - data parsing in different formats
    - sentence splitting
    - language detection
    - filtering of ill-formed sentences 
    - deduplication of repetitive contents
    - keep the original document boundaries
  
  Finally, the corpora are concatenated and further global deduplication among the corpora have been applied.
  The result is a medium-size biomedical corpus for Spanish composed of about 963M tokens. The table below shows some basic statistics of the individual cleaned corpora:
  
      
  | Name                                                                                    | No. tokens  | Description                                                                                                                          