readme: ---
  language: multilingual
  license: mit
  widget:
  - text: "and I cannot conceive the reafon why [MASK] hath"
  - text: "Täkäläinen sanomalehdistö [MASK] erit - täin"
  - text: "Det vore [MASK] häller nödvändigt att be"
  - text: "Comme, à cette époque [MASK] était celle de la"
  - text: "In [MASK] an atmosphärischen Nahrungsmitteln"
  ---
  
  # Historic Language Models (HLMs)
  
  ## Languages
  
  Our Historic Language Models Zoo contains support for the following languages - incl. their training data source:
  
  | Language | Training data | Size 
  | -------- | ------------- | ----
  | German   | [Europeana](http://www.europeana-newspapers.eu/)       | 13-28GB (filtered)
  | French   | [Europeana](http://www.europeana-newspapers.eu/)       | 11-31GB (filtered)
  | English  | [British Library](https://data.bl.uk/digbks/db14.html) | 24GB (year filtered)
  | Finnish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.2GB
  | Swedish  | [Europeana](http://www.europeana-newspapers.eu/)       | 1.1GB
  
  ## Models
  
  At the moment, the following models are available on the model hub:
  
  | Model identifier                              | Model Hub link
  | --------------------------------------------- | --------------------------------------------------------------------------
  | `dbmdz/bert-base-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)
  | `dbmdz/bert-base-historic-english-cased`      | [here](https://huggingface.co/dbmdz/bert-base-historic-english-cased)
  | `dbmdz/bert-base-finnish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-finnish-europeana-cased)
  | `dbmdz/bert-base-swedish-europeana-cased`     | [here](https://huggingface.co/dbmdz/bert-base-swedish-europeana-cased)
  
  We also released smaller models for the multilingual model:
  
  | Model identifier | Model Hub link
  | ----------------------------------------------- | ---------------------------------------------------------------------------
  | `dbmdz/bert-tiny-historic-multilingual-cased`   | [here](https://huggingface.co/dbmdz/bert-tiny-historic-multilingual-cased)
  | `dbmdz/bert-mini-historic-multilingual-cased`   | [here](https://huggingface.co/dbmdz/bert-mini-historic-multilingual-cased)
  | `dbmdz/bert-small-historic-multilingual-cased`  | [here](https://huggingface.co/dbmdz/bert-small-historic-multilingual-cased)
  | `dbmdz/bert-medium-historic-multilingual-cased` | [here](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased)
  
  **Notice**: We have released language models for Historic German and French trained on more noisier data earlier - see
  [this repo](https://github.com/stefan-it/europeana-bert) for more information:
  
  | Model identifier                              | Model Hub link
  | --------------------------------------------- | --------------------------------------------------------------------------
  | `dbmdz/bert-base-german-europeana-cased`      | [here](https://huggingface.co/dbmdz/bert-base-german-europeana-cased)
  | `dbmdz/bert-base-french-europeana-cased`      | [here](https://huggingface.co/dbmdz/bert-base-french-europeana-cased)
  
  # Corpora Stats
  
  ## German Europeana Corpus
  
  We provide some statistics using different thresholds of ocr confidences, in order to shrink down the corpus size
  and use less-noisier data:
  
  | OCR confidence | Size
  | -------------- | ----
  | **0.60**       | 28GB
  | 0.65           | 18GB
  | 0.70           | 13GB
  
  For the final corpus we use a OCR confidence of 0.6 (28GB). The following plot shows a tokens per year distribution:
  
  ![German Europeana Corpus Stats](stats/figures/german_europeana_corpus_stats.png)
  
  ## French Europeana Corpus
  
  Like German, we use different ocr confidence thresholds:
  
  | OCR confidence | Size
  | -------------- | ----
  | 0.60           | 31GB
  | 0.65           | 27GB
  | **0.70**       | 27GB
  | 0.75           | 23GB
  | 0.80           | 11GB
  
  For the final corpus we use a OCR confidence of 0.7 (27GB). The following plot shows a tokens per year distribution:
  
  ![French Europeana Corpus Stats](stats/figures/french_europeana_corpus_stats.png)
  
  ## British Library Corpus
  
  Metadata is taken from [here](https://data.bl.uk/digbks/DB21.html). Stats incl. year filtering:
  
  | Years             | Size
  | ----------------- | ----
  | ALL               | 24GB
  | >= 1800 && < 1900 | 24GB
  
  We use the year filtered variant. The following plot shows a tokens per year distribution:
  
  ![British Library Corpus Stats](stats/figures/bl_corpus_stats.png)
  
  ## Finnish Europeana Corpus
  
  | OCR confidence | Size
  | -------------- | ----
  | 0.60           | 1.2GB
  
  The following plot shows a tokens per year distribution:
  
  ![Finnish Europeana Corpus Stats](stats/figures/finnish_europeana_corpus_stats.png)
  
  ## Swedish Europeana Corpus
  
  | OCR confidence | Size
  | -------------- | ----
  | 0.60           | 1.1GB
  
  The following plot shows a tokens per year distribution:
  
  ![Swedish Europeana Corpus Stats](stats/figures/swedish_europeana_corpus_stats.png)
  
  ## All Corpora
  
  The following plot shows a