|
|
--- |
|
|
tags: |
|
|
- generated_from_trainer |
|
|
model-index: |
|
|
- name: bertlawbr |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
|
|
# bertlawbr |
|
|
|
|
|
This model is a fine-tuned version of [](https://huggingface.co/) on the None dataset. |
|
|
It achieves the following results on the evaluation set: |
|
|
- Loss: 1.0495 |
|
|
|
|
|
## Model description |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Intended uses & limitations |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Training and evaluation data |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
### Training hyperparameters |
|
|
|
|
|
The following hyperparameters were used during training: |
|
|
- learning_rate: 0.0001 |
|
|
- train_batch_size: 16 |
|
|
- eval_batch_size: 16 |
|
|
- seed: 42 |
|
|
- gradient_accumulation_steps: 8 |
|
|
- total_train_batch_size: 128 |
|
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06 |
|
|
- lr_scheduler_type: linear |
|
|
- lr_scheduler_warmup_steps: 10000 |
|
|
- num_epochs: 20.0 |
|
|
- mixed_precision_training: Native AMP |
|
|
|
|
|
### Training results |
|
|
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
|
|:-------------:|:-----:|:------:|:---------------:| |
|
|
| 6.1291 | 0.22 | 2500 | 5.9888 | |
|
|
| 4.8604 | 0.44 | 5000 | 4.4841 | |
|
|
| 3.3321 | 0.66 | 7500 | 3.1190 | |
|
|
| 2.7579 | 0.87 | 10000 | 2.6089 | |
|
|
| 2.4135 | 1.09 | 12500 | 2.3029 | |
|
|
| 2.2136 | 1.31 | 15000 | 2.1244 | |
|
|
| 2.0735 | 1.53 | 17500 | 1.9931 | |
|
|
| 1.9684 | 1.75 | 20000 | 1.8878 | |
|
|
| 1.891 | 1.97 | 22500 | 1.8077 | |
|
|
| 1.8215 | 2.18 | 25000 | 1.7487 | |
|
|
| 1.7577 | 2.4 | 27500 | 1.6875 | |
|
|
| 1.7113 | 2.62 | 30000 | 1.6444 | |
|
|
| 1.6776 | 2.84 | 32500 | 1.6036 | |
|
|
| 1.6203 | 3.06 | 35000 | 1.5608 | |
|
|
| 1.6018 | 3.28 | 37500 | 1.5293 | |
|
|
| 1.5602 | 3.5 | 40000 | 1.5044 | |
|
|
| 1.5429 | 3.71 | 42500 | 1.4753 | |
|
|
| 1.5148 | 3.93 | 45000 | 1.4472 | |
|
|
| 1.4786 | 4.15 | 47500 | 1.4302 | |
|
|
| 1.4653 | 4.37 | 50000 | 1.4128 | |
|
|
| 1.4496 | 4.59 | 52500 | 1.3991 | |
|
|
| 1.4445 | 4.81 | 55000 | 1.3943 | |
|
|
| 1.5114 | 5.02 | 57500 | 1.4551 | |
|
|
| 1.5054 | 5.24 | 60000 | 1.4525 | |
|
|
| 1.4817 | 5.46 | 62500 | 1.4259 | |
|
|
| 1.48 | 5.68 | 65000 | 1.4077 | |
|
|
| 1.4526 | 5.9 | 67500 | 1.3912 | |
|
|
| 1.4272 | 6.12 | 70000 | 1.3726 | |
|
|
| 1.4078 | 6.34 | 72500 | 1.3596 | |
|
|
| 1.399 | 6.55 | 75000 | 1.3450 | |
|
|
| 1.386 | 6.77 | 77500 | 1.3328 | |
|
|
| 1.3704 | 6.99 | 80000 | 1.3192 | |
|
|
| 1.3538 | 7.21 | 82500 | 1.3131 | |
|
|
| 1.3468 | 7.43 | 85000 | 1.2916 | |
|
|
| 1.323 | 7.65 | 87500 | 1.2871 | |
|
|
| 1.322 | 7.86 | 90000 | 1.2622 | |
|
|
| 1.2956 | 8.08 | 92500 | 1.2624 | |
|
|
| 1.2869 | 8.3 | 95000 | 1.2547 | |
|
|
| 1.2763 | 8.52 | 97500 | 1.2404 | |
|
|
| 1.275 | 8.74 | 100000 | 1.2305 | |
|
|
| 1.2709 | 8.96 | 102500 | 1.2301 | |
|
|
| 1.2514 | 9.18 | 105000 | 1.2179 | |
|
|
| 1.2563 | 9.39 | 107500 | 1.2134 | |
|
|
| 1.2487 | 9.61 | 110000 | 1.2111 | |
|
|
| 1.2337 | 9.83 | 112500 | 1.2041 | |
|
|
| 1.3215 | 10.05 | 115000 | 1.2879 | |
|
|
| 1.3364 | 10.27 | 117500 | 1.2850 | |
|
|
| 1.3286 | 10.49 | 120000 | 1.2779 | |
|
|
| 1.3202 | 10.7 | 122500 | 1.2730 | |
|
|
| 1.3181 | 10.92 | 125000 | 1.2651 | |
|
|
| 1.2952 | 11.14 | 127500 | 1.2544 | |
|
|
| 1.2889 | 11.36 | 130000 | 1.2506 | |
|
|
| 1.2747 | 11.58 | 132500 | 1.2339 | |
|
|
| 1.2729 | 11.8 | 135000 | 1.2277 | |
|
|
| 1.2699 | 12.02 | 137500 | 1.2201 | |
|
|
| 1.2508 | 12.23 | 140000 | 1.2163 | |
|
|
| 1.2438 | 12.45 | 142500 | 1.2091 | |
|
|
| 1.2445 | 12.67 | 145000 | 1.2003 | |
|
|
| 1.2314 | 12.89 | 147500 | 1.1957 | |
|
|
| 1.2188 | 13.11 | 150000 | 1.1843 | |
|
|
| 1.2071 | 13.33 | 152500 | 1.1805 | |
|
|
| 1.2123 | 13.54 | 155000 | 1.1766 | |
|
|
| 1.2016 | 13.76 | 157500 | 1.1661 | |
|
|
| 1.2079 | 13.98 | 160000 | 1.1625 | |
|
|
| 1.1884 | 14.2 | 162500 | 1.1525 | |
|
|
| 1.177 | 14.42 | 165000 | 1.1419 | |
|
|
| 1.1793 | 14.64 | 167500 | 1.1454 | |
|
|
| 1.173 | 14.85 | 170000 | 1.1379 | |
|
|
| 1.1502 | 15.07 | 172500 | 1.1371 | |
|
|
| 1.1504 | 15.29 | 175000 | 1.1295 | |
|
|
| 1.146 | 15.51 | 177500 | 1.1203 | |
|
|
| 1.1487 | 15.73 | 180000 | 1.1137 | |
|
|
| 1.1329 | 15.95 | 182500 | 1.1196 | |
|
|
| 1.1259 | 16.17 | 185000 | 1.1075 | |
|
|
| 1.1287 | 16.38 | 187500 | 1.1037 | |
|
|
| 1.126 | 16.6 | 190000 | 1.1042 | |
|
|
| 1.1199 | 16.82 | 192500 | 1.0953 | |
|
|
| 1.1072 | 17.04 | 195000 | 1.0885 | |
|
|
| 1.1043 | 17.26 | 197500 | 1.0877 | |
|
|
| 1.1007 | 17.48 | 200000 | 1.0835 | |
|
|
| 1.0879 | 17.69 | 202500 | 1.0819 | |
|
|
| 1.1 | 17.91 | 205000 | 1.0744 | |
|
|
| 1.0863 | 18.13 | 207500 | 1.0774 | |
|
|
| 1.087 | 18.35 | 210000 | 1.0759 | |
|
|
| 1.0755 | 18.57 | 212500 | 1.0618 | |
|
|
| 1.0832 | 18.79 | 215000 | 1.0628 | |
|
|
| 1.0771 | 19.01 | 217500 | 1.0611 | |
|
|
| 1.0703 | 19.22 | 220000 | 1.0555 | |
|
|
| 1.069 | 19.44 | 222500 | 1.0552 | |
|
|
| 1.0706 | 19.66 | 225000 | 1.0509 | |
|
|
| 1.0633 | 19.88 | 227500 | 1.0465 | |
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- Transformers 4.12.5 |
|
|
- Pytorch 1.10.1+cu113 |
|
|
- Datasets 1.17.0 |
|
|
- Tokenizers 0.10.3 |
|
|
|
|
|
## Citing & Authors |
|
|
|
|
|
If you use our work, please cite: |
|
|
|
|
|
``` |
|
|
@incollection{Viegas_2023, |
|
|
doi = {10.1007/978-3-031-36805-9_24}, |
|
|
url = {https://doi.org/10.1007%2F978-3-031-36805-9_24}, |
|
|
year = 2023, |
|
|
publisher = {Springer Nature Switzerland}, |
|
|
pages = {349--365}, |
|
|
author = {Charles F. O. Viegas and Bruno C. Costa and Renato P. Ishii}, |
|
|
title = {{JurisBERT}: A New Approach that~Converts a~Classification Corpus into~an~{STS} One}, |
|
|
booktitle = {Computational Science and Its Applications {\textendash} {ICCSA} 2023} |
|
|
} |
|
|
``` |
|
|
|
