readme: ---
  language:
  - es
  tags:
  - albert
  - spanish
  - OpenCENIA
  datasets:
  - large_spanish_corpus
  ---
  
  # ALBERT XXLarge Spanish
  
  This is an [ALBERT](https://github.com/google-research/albert) model trained on a [big spanish corpora](https://github.com/josecannete/spanish-corpora).
  The model was trained on a single TPU v3-8 with the following hyperparameters and steps/time:
  - LR: 0.0003125
  - Batch Size: 128
  - Warmup ratio: 0.00078125
  - Warmup steps: 3125
  - Goal steps: 4000000
  - Total steps: 1650000
  - Total training time (aprox): 70.7 days.
  
  ## Training loss
  
  ![https://drive.google.com/uc?export=view&id=1a9MHsk-QwBuCMtyDyRvZ5mv9Mzl2dWCn](https://drive.google.com/uc?export=view&id=1a9MHsk-QwBuCMtyDyRvZ5mv9Mzl2dWCn)