readme: ---
  language: it
  tags:
  - DISTILbert
  - Italian
  license: mit
  widget:
  - text: Vado al [MASK] a fare la spesa
  - text: Vado al parco a guardare le [MASK]
  - text: Il cielo è [MASK] di stelle.
  ---
  
  
  # BERTino: an Italian DistilBERT model
  This repository hosts BERTino, an Italian DistilBERT model pre-trained by
  [indigo.ai](https://indigo.ai/en/)
  on a large general-domain Italian corpus. BERTino is task-agnostic and can be 
  fine-tuned for every downstream task.
  ### Corpus
  The pre-training corpus that we used is the union of the
  [Paisa](https://www.corpusitaliano.it/) and
  [ItWaC](https://corpora.dipintra.it/public/run.cgi/corp_info?corpname=itwac_full)
  corpora. The final corpus counts 14 millions of sentences for a total of 12 GB
  of text.
  ### Downstream Results
  To validate the pre-training that we conducted, we evaluated BERTino on the
  [Italian ParTUT](https://universaldependencies.org/treebanks/it_partut/index.html),
  [Italian ISDT](https://universaldependencies.org/treebanks/it_isdt/index.html),
  [Italian WikiNER](https://figshare.com/articles/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500)
  and multi-class sentence classification tasks. We report for comparison results
  obtained by the [teacher model](https://huggingface.co/dbmdz/bert-base-italian-xxl-uncased)
  fine-tuned in the same tasks and for the same number of epochs.
  
  **Italian ISDT:**
  
  | Model        | F1 score | Fine-tuning time | Evaluation time |
  |--------------|----------|------------------|-----------------|
  | BERTino      | 0,9801   | 9m, 4s           | 3s              |
  | Teacher      | 0,983    | 16m, 28s         | 5s              |
  
  **Italian ParTUT:**
  
  | Model        | F1 score | Fine-tuning time | Evaluation time |
  |--------------|----------|------------------|-----------------|
  | BERTino      | 0,9268   | 1m, 18s           | 1s             |
  | Teacher      | 0,9688    | 2m, 18s         | 1s              |
  
  **Italian WikiNER:**
  
  | Model        | F1 score | Fine-tuning time | Evaluation time |
  |--------------|----------|------------------|-----------------|
  | BERTino      | 0,9038  | 35m, 35s           | 3m, 1s             |
  | Teacher      | 0,9178    | 67m, 8s         | 5m, 16s              |
  
  **Multi-class sentence classification:**
  
  | Model        | F1 score | Fine-tuning time | Evaluation time |
  |--------------|----------|------------------|-----------------|
  | BERTino      | 0,7788   | 4m, 40s           | 6s             |
  | Teacher      | 0,7986    | 8m, 52s         | 9s              |
  
  
  