readme: ---
  language: 
  - pt
  tags:
  - generated_from_trainer
  datasets:
  - pierreguillou/lener_br_finetuning_language_model
  model-index:
  - name: checkpoints
    results:
    - task:
        name: Fill Mask
        type: fill-mask
      dataset:
        name: pierreguillou/lener_br_finetuning_language_model
        type: pierreguillou/lener_br_finetuning_language_model
      metrics:
      - name: Loss
        type: loss
        value: 1.127950
  widget:
  - text: "Com efeito, se tal fosse possível, o Poder [MASK] – que não dispõe de função legislativa – passaria a desempenhar atribuição que lhe é institucionalmente estranha (a de legislador positivo), usurpando, desse modo, no contexto de um sistema de poderes essencialmente limitados, competência que não lhe pertence, com evidente transgressão ao princípio constitucional da separação de poderes."
  ---
  
  ## (BERT large) Language modeling in the legal domain in Portuguese (LeNER-Br)
  
  **bert-large-cased-pt-lenerbr** is a Language Model in the legal domain in Portuguese that was finetuned on 20/12/2021 in Google Colab from the model [BERTimbau large](https://huggingface.co/neuralmind/bert-large-portuguese-cased) on the dataset [LeNER-Br language modeling](https://huggingface.co/datasets/pierreguillou/lener_br_finetuning_language_model) by using a MASK objective.
  
  You can check as well the [version base of this model](https://huggingface.co/pierreguillou/bert-base-cased-pt-lenerbr).
    
  ## Widget & APP
  
  You can test this model into the widget of this page.
  
  ## Blog post
  
  This language model is used to get a NER model on the Portuguese judicial domain. You can check the fine-tuned NER model at [pierreguillou/ner-bert-large-cased-pt-lenerbr](https://huggingface.co/pierreguillou/ner-bert-large-cased-pt-lenerbr).
  
  All informations and links are in this blog post: [NLP | Modelos e Web App para Reconhecimento de Entidade Nomeada (NER) no domínio jurídico brasileiro](https://medium.com/@pierre_guillou/nlp-modelos-e-web-app-para-reconhecimento-de-entidade-nomeada-ner-no-dom%C3%ADnio-jur%C3%ADdico-b658db55edfb) (29/12/2021)
  
  ## Using the model for inference in production
  ````
  # install pytorch: check https://pytorch.org/
  # !pip install transformers 
  from transformers import AutoTokenizer, AutoModelForMaskedLM
  
  tokenizer = AutoTokenizer.from_pretrained("pierreguillou/bert-large-cased-pt-lenerbr")
  model = AutoModelForMaskedLM.from_pretrained("pierreguillou/bert-large-cased-pt-lenerbr")
  ````
  
  ## Training procedure
  
  ## Notebook
  
  The notebook of finetuning ([Finetuning_language_model_BERtimbau_LeNER_Br.ipynb](https://github.com/piegu/language-models/blob/master/Finetuning_language_model_BERtimbau_LeNER_Br.ipynb)) is in github.
  
  ### Training results
  
  ````
  Num examples = 3227
  Num Epochs = 5
  Instantaneous batch size per device = 2
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 4
  Total optimization steps = 2015
  
  Step	Training Loss	Validation Loss
  100   1.616700      1.366015
  200   1.452000      1.312473
  300   1.431100      1.253055
  400   1.407500      1.264705
  500   1.301900      1.243277
  600   1.317800      1.233684
  700   1.319100      1.211826
  800   1.303800      1.190818
  900   1.262800      1.171898
  1000  1.235900      1.146275
  1100  1.221900      1.149027
  1200  1.226200      1.127950
  1300  1.201700      1.172729
  1400  1.198200      1.145363
  ````