readme: ---
  language: en
  pipeline_tag: fill-mask
  license: cc-by-sa-4.0
  thumbnail: https://i.ibb.co/p3kQ7Rw/Screenshot-2020-10-06-at-12-16-36-PM.png
  tags:
  - legal
  widget:
  - text: "The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of police."
  ---
  
  # LEGAL-BERT: The Muppets straight out of Law School
  
  <img align="left" src="https://i.ibb.co/p3kQ7Rw/Screenshot-2020-10-06-at-12-16-36-PM.png" width="100"/> 
  
  LEGAL-BERT is a family of BERT models for the legal domain, intended to assist legal NLP research, computational law, and legal technology applications. To pre-train the different variations of LEGAL-BERT, we collected 12 GB of diverse English legal text from several fields (e.g., legislation, court cases,  contracts) scraped from publicly available resources. Sub-domain variants (CONTRACTS-, EURLEX-, ECHR-) and/or general LEGAL-BERT perform better than using BERT out of the box for domain-specific tasks.<br>
  This is the light-weight version of BERT-BASE (33% the size of BERT-BASE) pre-trained from scratch on legal data, which achieves comparable performance to larger models, while being much more efficient (approximately 4 times faster) with a smaller environmental footprint.
  <br/><br/>
  
  ---
  
  I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras and I. Androutsopoulos. "LEGAL-BERT: The Muppets straight out of Law School". In Findings of Empirical Methods in Natural Language Processing (EMNLP 2020) (Short Papers), to be held online, 2020. (https://aclanthology.org/2020.findings-emnlp.261)
  
  ---
  
  ## Pre-training corpora
  
  The pre-training corpora of LEGAL-BERT include:
  
  * 116,062 documents of EU legislation, publicly available from EURLEX (http://eur-lex.europa.eu), the repository of EU Law running under the EU Publication Office.
      
  * 61,826 documents of UK legislation, publicly available from the UK legislation portal (http://www.legislation.gov.uk).
      
  * 19,867 cases from the European Court of Justice (ECJ), also available from EURLEX.
      
  * 12,554 cases from HUDOC, the repository of the European Court of Human Rights (ECHR) (http://hudoc.echr.coe.int/eng).
      
  * 164,141 cases from various courts across the USA, hosted in the Case Law Access Project portal (https://case.law).
      
  * 76,366 US contracts from EDGAR, the database of US Securities and Exchange Commission (SECOM) (https://www.sec.gov/edgar.shtml).
  
  ## Pre-training details
  
  * We trained BERT using the official code provided in Google BERT's GitHub repository (https://github.com/google-research/bert).
  * We released a model similar to the English BERT-BASE model (12-layer, 768-hidden, 12-heads, 110M parameters).
  * We chose to follow the same training set-up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e-4.
  * We were able to use a single Google Cloud TPU v3-8 provided for free from [TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc), while also utilizing [GCP research credits](https://edu.google.com/programs/credits/research). Huge thanks to both Google programs for supporting us!
  
  ## Models list
  
  | Model name          | Model Path                            | Training corpora    |
  | ------------------- | ------------------------------------  | ------------------- |
  | CONTRACTS-BERT-BASE | `nlpaueb/bert-base-uncased-contracts` | US contracts        |
  | EURLEX-BERT-BASE    | `nlpaueb/bert-base-uncased-eurlex`    | EU legislation      |
  | ECHR-BERT-BASE      | `nlpaueb/bert-base-uncased-echr`      | ECHR cases          |
  | LEGAL-BERT-BASE *     | `nlpaueb/legal-bert-base-uncased`     | All                 |
  | LEGAL-BERT-SMALL    | `nlpaueb/legal-bert-small-uncased`    | All                 |
  
  \* LEGAL-BERT-BASE is the model referred to as LEGAL-BERT-SC in Chalkidis et al. (2020); a model trained from scratch in the legal corpora mentioned below using a newly created vocabulary by a sentence-piece tokenizer trained on the very same corpora.
  
  \*\* As many of you expressed interest in the LEGAL-BERT-FP models (those relying on the original BERT-BASE checkpoint), they have been released in Archive.org (https://archive.org/details/legal_bert_fp), as these models are secondary and possibly only interesting for those who aim to dig deeper in the open questions of Chalkidis et al. (2020).
  
  ## Load Pretrained Model
  
  ```python
  from transformers import AutoTokenizer, AutoModel
  
  tokenizer = AutoTokenizer.from_pretrained("nlpaueb/legal-bert-small-uncased")
  model = AutoModel.from_pretrained("nlpaueb/legal-bert-small-uncased")
  ```
  
  ## Use LEGAL-BERT variants as Language Models
  
  | Corpus                             | Model                               | Masked token | Predictions  |
  | --------------------------------- | ---------------------------------- | ------------ | ------------ |
  |  | **BERT-BASE-UNCASED**                 |
  | (Contracts) | This [MASK] Agreement is between General Motors and John Murray . | employment | ('new', '0.