readme: Model card for jurBERT-base
  
  ---
  language: 
  - ro
  ---
  
  # jurBERT-base
  
  
  ## Pretrained juridical BERT model for Romanian 
  
  BERT Romanian juridical model trained using a masked language modeling (MLM) and next sentence prediction (NSP) objective. 
  It was introduced in this [paper](https://aclanthology.org/2021.nllp-1.8/). Two BERT models were released: **jurBERT-base** and **jurBERT-large**, all versions uncased.
  
  | Model          | Weights   |   L    |   H    |    A   | MLM accuracy   | NSP accuracy   |
  |----------------|:---------:|:------:|:------:|:------:|:--------------:|:--------------:|
  | *jurBERT-base*    | *111M*      | *12*     | *768*    | *12*     | *0.8936*         | *0.9923*         |
  | jurBERT-large   | 337M      | 24     | 1024   | 24     | 0.9005         | 0.9929         |
  
  
  
  
  All models are available:
  
  * [jurBERT-base](https://huggingface.co/readerbench/jurBERT-base)
  * [jurBERT-large](https://huggingface.co/readerbench/jurBERT-large)
  
  
  
  #### How to use
  
  ```python
  # tensorflow
  from transformers import AutoModel, AutoTokenizer, TFAutoModel
  tokenizer = AutoTokenizer.from_pretrained("readerbench/jurBERT-base")
  model = TFAutoModel.from_pretrained("readerbench/jurBERT-base")
  inputs = tokenizer("exemplu de propoziție", return_tensors="tf")
  outputs = model(inputs)
  
  
  # pytorch
  from transformers import AutoModel, AutoTokenizer, AutoModel
  tokenizer = AutoTokenizer.from_pretrained("readerbench/jurBERT-base")
  model = AutoModel.from_pretrained("readerbench/jurBERT-base")
  inputs = tokenizer("exemplu de propoziție", return_tensors="pt")
  outputs = model(**inputs)
  ```
  
  
  ## Datasets
  
  The model is trained on a private corpus (that can nevertheless be rented for a fee), that is comprised of all the final ruling, containing both civil and criminal cases, published by any Romanian civil court between 2010 and 2018. Validation is performed on two other datasets, RoBanking and BRDCases. We extracted from RoJur common types of cases pertinent to the banking domain (e.g. administration fee litigations, enforcement appeals), kept only the summary of the arguments provided by both the plaitiffs and the defendants and the final verdict (in the form of a boolean value) to build RoBanking. BRDCases represents a collection of cases in which BRD Groupe Société Générale Romania was directly involved. 
  
  | Corpus    | Scope        |Entries    |  Size (GB)|
  |-----------|:------------:|:---------:|:---------:|
  | RoJur     | pre-training | 11M       | 160       |
  | RoBanking | downstream   | 108k      | -         |
  | BRDCases  | downstream   | 149       | -         |
  
  
  ## Downstream performance
  
  We report Mean AUC and Std AUC on the task of predicting the outcome of a case. 
  
  ### Results on RoBanking using only the plea of the plaintiff.
  
  | Model              | Mean AUC | Std AUC  |
  |--------------------|:--------:|:--------:|
  | CNN                | 79.60    | -        |
  | BI-LSTM            | 80.99    | 0.26     |
  | RoBERT-small       | 70.54    | 0.28     |
  | RoBERT-base        | 79.74    | 0.21     |
  | RoBERT-base + hf   | 79.82    | 0.11     |
  | RoBERT-large       | 76.53    | 5.43     |
  | *jurBERT-base*     | **81.47**| **0.18** |
  | *jurBERT-base + hf*| *81.40*  | *0.18*   |
  | jurBERT-large      | 78.38    | 1.77     |
  
  ### Results on RoBanking using pleas from both the plaintiff and defendant.
  
  | Model               | Mean AUC | Std AUC  |
  |---------------------|:--------:|:--------:|
  | BI-LSTM             | 84.60    | 0.59     |
  | RoBERT-base         | 84.40    | 0.26     |
  | RoBERT-base + hf    | 84.43    | 0.15     |
  | *jurBERT-base*      | *86.63*  | *0.18*   |
  | *jurBERT-base + hf* | **86.73**| **0.22** |
  | jurBERT-large       | 82.04    | 0.64     |
  
  ### Results on BRDCases
  
  | Model               | Mean AUC | Std AUC  |
  |---------------------|:--------:|:--------:|
  | SVM with SK         | 57.72    | 2.15     |
  | RoBERT-base         | 53.24    | 1.76     |
  | RoBERT-base + hf    | 55.40    | 0.96     |
  | *jurBERT-base*      | *59.65*  | *1.16*   |
  | *jurBERT-base + hf* | **61.46**| **1.76** |
  
  For complete results and discussion please refer to the [paper](https://aclanthology.org/2021.nllp-1.8/).
  
  ### BibTeX entry and citation info
  
  ```bibtex
  @inproceedings{masala2021jurbert,
    title={jurBERT: A Romanian BERT Model for Legal Judgement Prediction},
    author={Masala, Mihai and Iacob, Radu Cristian Alexandru and Uban, Ana Sabina and Cidota, Marina and Velicu, Horia and Rebedea, Traian and Popescu, Marius},
    booktitle={Proceedings of the Natural Legal Language Processing Workshop 2021},
    pages={86--94},
    year={2021}
  }
  ```
  
  