readme: ---
  language: en
  tags:
  - newspapers
  - library
  - historic
  - glam
  - mdma
  license: mit
  metrics:
  - pseudo-perplexity
  widget:
  - text: "1820 [DATE] We received a letter from [MASK] Majesty."
  - text: "1850 [DATE] We received a letter from [MASK] Majesty."
  - text: "[MASK] [DATE] The Franco-Prussian war is a matter of great concern."
  - text: "[MASK] [DATE] The Schleswig war is a matter of great concern."
  
  ---
  **MODEL CARD UNDER CONSTRUCTION, ETA END OF NOVEMBER**
  
  <img src="https://upload.wikimedia.org/wikipedia/commons/5/5b/NCI_peas_in_pod.jpg" alt="erwt" width="200" >
   
  # ERWT-year
  
  🌺ERWT\* a language model that (🤭 maybe 🤫) knows more about history than you...🌺
  
  ERWT is a fine-tuned [`distilbert-base-uncased`](https://huggingface.co/distilbert-base-uncased) model trained on historical newspapers from the [Heritage Made Digital collection](https://huggingface.co/datasets/davanstrien/hmd-erwt-training).
  
   We trained a model based on a combination of text and **temporal metadata** (i.e. year information).
  
  ERWT performs [**time-sensitive masked language modelling**](#historical-language-change-herhis-majesty-%F0%9F%91%91) or [**date prediction**]((#date-prediction-pub-quiz-with-lms-%F0%9F%8D%BB)).
  
  This model is served by [Kaspar von Beelen](https://huggingface.co/Kaspar) and [Daniel van Strien](https://huggingface.co/davanstrien), *"Improving AI, one pea at a time"*.
  
  If these models happen to be useful, please cite our working paper.
  
  ```
  @misc{https://doi.org/10.48550/arxiv.2211.10086,
    doi = {10.48550/ARXIV.2211.10086},
    url = {https://arxiv.org/abs/2211.10086},
    author = {Beelen, Kaspar and van Strien, Daniel},
    keywords = {Computation and Language (cs.CL), Digital Libraries (cs.DL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Metadata Might Make Language Models Better},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}}
  ```
  
  \*ERWT is dutch for PEA.
  
  # Overview
  
  - [Introduction: Repent Now 😇](#introductory-note-repent-now-%F0%9F%98%87)
  - [Background: MDMA to the rescue 🙂](#background-mdma-to-the-rescue-%F0%9F%99%82)
  - [Intended Use: LMs as History Machines 🚂](#intended-use-lms-as-history-machines)
     - [Historical Language Change: Her/His Majesty? 👑](#historical-language-change-herhis-majesty-%F0%9F%91%91)
     - [Date Prediction: Pub Quiz with LMs 🍻](#date-prediction-pub-quiz-with-lms-%F0%9F%8D%BB)
  - [Limitations: Not all is well 😮](#limitations-not-all-is-well-%F0%9F%98%AE)
      - [Training Data](#training-data)
      - [Training Routine](#training-routine)
  - [Data Description](#data-description)
  - [Evaluation: 🤓 In case you care to count 🤓](#evaluation-%F0%9F%A4%93-in-case-you-care-to-count-%F0%9F%A4%93)
   
     
  
  ## Introductory Note: Repent Now. 😇
  
  The ERWT models are trained for **experimental purposes**. 
  
  Please consult the [**limitations**](#limitations-not-all-is-well-%F0%9F%98%AE) section (seriously before using the models. Seriously, read this section, **we don't repent in public just for fun**). 
  
  If you can't get enough of these neural peas and crave some more. In that case, you can consult our working paper ["Metadata Might Make Language Models Better"](https://arxiv.org/abs/2211.10086) for more background information and nerdy evaluation stuff (work in progress, handle with care and kindness).
  
  ## Background: MDMA to the rescue. 🙂
  
  ERWT was created using a **M**eta**D**ata **M**asking **A**pproach (or **MDMA** 💊), a scenario in which we train a Masked Language Model (MLM) on text and metadata simultaneously. Our intuition was that incorporating metadata (information that *describes* a text but and is not part of the content) may make language models "better", or at least make them more **sensitive** to historical, political and geographical aspects of language use. We mainly use temporal, political and geographical metadata.
  
  ERWT is a [`distilbert-base-uncased`](https://huggingface.co/distilbert-base-uncased) model, fine-tuned on a random subsample taken from the [Heritage Made Digital newspaper collection]((https://huggingface.co/datasets/davanstrien/hmd-erwt-training)). The training data comprises around half a billion words. 
  
  To unleash the power of MDMA, we adapted to the training routine mainly by fidgeting with the input data. 
  
  When preprocessing the text, we prepended each segment of hundred words with a time stamp (year of publication) and a special `[DATE]` token. 
  
  
  The snippet below, taken from the [Londonderry Sentinel]:(https://www.britishnewspaperarchive.co.uk/viewer/bl/0001480/18700722/014/0002)...
  ```
  Every scrap of intelligence relative to the war between France and Prussia is now read with interest.
  ```
   
  ... would be formatted as:
        
  ```python
  "1870 [DATE] Every scrap of intelligence relative to the war between France and Prussia is now read with interest."
  ```
  
  These text chunks are then forwarded to the data collator and eventually the language model.
  
  Exposed to the token