readme: ---
  language: 
    - en
  datasets:
    - pile-of-law/pile-of-law
  pipeline_tag: fill-mask
  ---
  
  # Pile of Law BERT large model 2 (uncased)
  Pretrained model on English language legal and administrative text using the [RoBERTa](https://arxiv.org/abs/1907.11692) pretraining objective. This model was trained with the same setup as [pile-of-law/legalbert-large-1.7M-1](https://huggingface.co/pile-of-law/legalbert-large-1.7M-1), but with a different seed.
  
  ## Model description
  Pile of Law BERT large 2 is a transformers model with the [BERT large model (uncased)](https://huggingface.co/bert-large-uncased) architecture pretrained on the [Pile of Law](https://huggingface.co/datasets/pile-of-law/pile-of-law), a dataset consisting of ~256GB of English language legal and administrative text for language model pretraining.
  
  ## Intended uses & limitations
  You can use the raw model for masked language modeling or fine-tune it for a downstream task. Since this model was pretrained on a English language legal and administrative text corpus, legal downstream tasks will likely be more in-domain for this model.
  
  ## How to use
  You can use the model directly with a pipeline for masked language modeling:
  ```python
  >>> from transformers import pipeline
  >>> pipe = pipeline(task='fill-mask', model='pile-of-law/legalbert-large-1.7M-2')
  >>> pipe("An [MASK] is a request made after a trial by a party that has lost on one or more issues that a higher court review the decision to determine if it was correct.")
  
  [{'sequence': 'an exception is a request made after a trial by a party that has lost on one or more issues that a higher court review the decision to determine if it was correct.', 
    'score': 0.5218929052352905, 
    'token': 4028, 
    'token_str': 'exception'}, 
    {'sequence': 'an appeal is a request made after a trial by a party that has lost on one or more issues that a higher court review the decision to determine if it was correct.', 
    'score': 0.11434809118509293, 
    'token': 1151, 
    'token_str': 'appeal'}, 
    {'sequence': 'an exclusion is a request made after a trial by a party that has lost on one or more issues that a higher court review the decision to determine if it was correct.', 
    'score': 0.06454459577798843, 
    'token': 5345, 
    'token_str': 'exclusion'}, 
    {'sequence': 'an example is a request made after a trial by a party that has lost on one or more issues that a higher court review the decision to determine if it was correct.', 
    'score': 0.043593790382146835, 
    'token': 3677, 
    'token_str': 'example'}, 
    {'sequence': 'an objection is a request made after a trial by a party that has lost on one or more issues that a higher court review the decision to determine if it was correct.', 
    'score': 0.03758585825562477, 
    'token': 3542, 
    'token_str': 'objection'}]
  ```
  
  Here is how to use this model to get the features of a given text in PyTorch:
  
  ```python
  from transformers import BertTokenizer, BertModel
  tokenizer = BertTokenizer.from_pretrained('pile-of-law/legalbert-large-1.7M-2')
  model = BertModel.from_pretrained('pile-of-law/legalbert-large-1.7M-2')
  text = "Replace me by any text you'd like."
  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)
  ```
  
  and in TensorFlow:
  
  ```python
  from transformers import BertTokenizer, TFBertModel
  tokenizer = BertTokenizer.from_pretrained('pile-of-law/legalbert-large-1.7M-2')
  model = TFBertModel.from_pretrained('pile-of-law/legalbert-large-1.7M-2')
  text = "Replace me by any text you'd like."
  encoded_input = tokenizer(text, return_tensors='tf')
  output = model(encoded_input)
  ```
  
  ## Limitations and bias
  Please see Appendix G of the Pile of Law paper for copyright limitations related to dataset and model use.
  
  This model can have biased predictions. In the following example where the model is used with a pipeline for masked language modeling, for the race descriptor of the criminal, the model predicts a higher score for "black" than "white".
  
  ```python
  >>> from transformers import pipeline
  >>> pipe = pipeline(task='fill-mask', model='pile-of-law/legalbert-large-1.7M-2')
  >>> pipe("The transcript of evidence reveals that at approximately 7:30 a. m. on January 22, 1973, the prosecutrix was awakened in her home in DeKalb County by the barking of the family dog, and as she opened her eyes she saw a [MASK] man standing beside her bed with a gun.", targets=["black", "white"])
  
  [{'sequence': 'the transcript of evidence reveals that at approximately 7 : 30 a. m. on january 22, 1973, the prosecutrix was awakened in her home in dekalb county by the barking of the family dog, and as she opened her eyes she saw a black man standing beside her bed with a gun.', 
    'score': 0.02685137465596199, 
    'token': 4311, 
    'token_str': 'black'}, 
    {'sequence': 'the transcript of evidence reveals that at approximately 7 : 30 a. m. on january 22, 1973, the prosecutrix was awakened in her home in dekalb county by the barking of the family dog, and as she opened her eyes she s