|
--- |
|
license: apache-2.0 |
|
base_model: allenai/longformer-base-4096 |
|
tags: |
|
- generated_from_trainer |
|
datasets: |
|
- essays_su_g |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: longformer-simple |
|
results: |
|
- task: |
|
name: Token Classification |
|
type: token-classification |
|
dataset: |
|
name: essays_su_g |
|
type: essays_su_g |
|
config: simple |
|
split: train[80%:100%] |
|
args: simple |
|
metrics: |
|
- name: Accuracy |
|
type: accuracy |
|
value: 0.8449255946993012 |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# longformer-simple |
|
|
|
This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co/allenai/longformer-base-4096) on the essays_su_g dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.6609 |
|
- Claim: {'precision': 0.6078710289236605, 'recall': 0.6151631477927063, 'f1-score': 0.6114953493918436, 'support': 4168.0} |
|
- Majorclaim: {'precision': 0.782967032967033, 'recall': 0.7946096654275093, 'f1-score': 0.7887453874538746, 'support': 2152.0} |
|
- O: {'precision': 0.934072084172823, 'recall': 0.9045089963147627, 'f1-score': 0.9190528634361235, 'support': 9226.0} |
|
- Premise: {'precision': 0.8725067166001791, 'recall': 0.8876832601673155, 'f1-score': 0.8800295615043522, 'support': 12073.0} |
|
- Accuracy: 0.8449 |
|
- Macro avg: {'precision': 0.7993542156659239, 'recall': 0.8004912674255735, 'f1-score': 0.7998307904465485, 'support': 27619.0} |
|
- Weighted avg: {'precision': 0.8461593157460915, 'recall': 0.8449255946993012, 'f1-score': 0.8454278324403368, 'support': 27619.0} |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 8 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 16 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Claim | Majorclaim | O | Premise | Accuracy | Macro avg | Weighted avg | |
|
|:-------------:|:-----:|:----:|:---------------:|:---------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:| |
|
| No log | 1.0 | 41 | 0.5690 | {'precision': 0.49395770392749244, 'recall': 0.23536468330134358, 'f1-score': 0.31881702957426067, 'support': 4168.0} | {'precision': 0.5330313325783315, 'recall': 0.6561338289962825, 'f1-score': 0.5882107894188711, 'support': 2152.0} | {'precision': 0.9200096957944491, 'recall': 0.82278343810969, 'f1-score': 0.8686845568461407, 'support': 9226.0} | {'precision': 0.777574153261386, 'recall': 0.9488942267870455, 'f1-score': 0.8547340147728121, 'support': 12073.0} | 0.7763 | {'precision': 0.6811432213904147, 'recall': 0.6657940442985903, 'f1-score': 0.6576115976530211, 'support': 27619.0} | {'precision': 0.7632992267425562, 'recall': 0.7762772004779318, 'f1-score': 0.7577517824653167, 'support': 27619.0} | |
|
| No log | 2.0 | 82 | 0.4430 | {'precision': 0.6068347710683477, 'recall': 0.43881957773512476, 'f1-score': 0.5093288777499304, 'support': 4168.0} | {'precision': 0.6947840260798696, 'recall': 0.7922862453531598, 'f1-score': 0.7403386886669561, 'support': 2152.0} | {'precision': 0.930324074074074, 'recall': 0.8712334706264904, 'f1-score': 0.8998096943915818, 'support': 9226.0} | {'precision': 0.8270298275479239, 'recall': 0.9255363207156465, 'f1-score': 0.8735146966854284, 'support': 12073.0} | 0.8236 | {'precision': 0.7647431746925538, 'recall': 0.7569689036076054, 'f1-score': 0.7557479893734742, 'support': 27619.0} | {'precision': 0.8180007808150275, 'recall': 0.823563488902567, 'f1-score': 0.8169621924766614, 'support': 27619.0} | |
|
| No log | 3.0 | 123 | 0.4280 | {'precision': 0.5555102040816327, 'recall': 0.6530710172744721, 'f1-score': 0.6003528892809882, 'support': 4168.0} | {'precision': 0.7618816682832201, 'recall': 0.7300185873605948, 'f1-score': 0.7456098718557191, 'support': 2152.0} | {'precision': 0.9472815190470575, 'recall': 0.8705831346195534, 'f1-score': 0.9073143179892686, 'support': 9226.0} | {'precision': 0.8730497618656594, 'recall': 0.8806427565642343, 'f1-score': 0.8768298214506619, 'support': 12073.0} | 0.8312 | {'precision': 0.7844307883193924, 'recall': 0.7835788739547136, 'f1-score': 0.7825267251441594, 'support': 27619.0} | {'precision': 0.8412645262496828, 'recall': 0.8312031572468228, 'f1-score': 0.8350654121763821, 'support': 27619.0} | |
|
| No log | 4.0 | 164 | 0.4198 | {'precision': 0.6521200866604766, 'recall': 0.5055182341650671, 'f1-score': 0.5695364238410595, 'support': 4168.0} | {'precision': 0.7789709172259508, 'recall': 0.8090148698884758, 'f1-score': 0.7937086847503988, 'support': 2152.0} | {'precision': 0.91675722668985, 'recall': 0.9143724257533059, 'f1-score': 0.9155632732797916, 'support': 9226.0} | {'precision': 0.85398810902633, 'recall': 0.9160937629421022, 'f1-score': 0.8839514066496164, 'support': 12073.0} | 0.8452 | {'precision': 0.8004590849006519, 'recall': 0.7862498231872379, 'f1-score': 0.7906899471302166, 'support': 27619.0} | {'precision': 0.8386466761572305, 'recall': 0.8452152503711213, 'f1-score': 0.8400311740436862, 'support': 27619.0} | |
|
| No log | 5.0 | 205 | 0.4471 | {'precision': 0.5814893617021276, 'recall': 0.6557101727447217, 'f1-score': 0.6163734776725303, 'support': 4168.0} | {'precision': 0.7235804416403786, 'recall': 0.8526951672862454, 'f1-score': 0.7828498293515358, 'support': 2152.0} | {'precision': 0.9300457436126297, 'recall': 0.9035334923043572, 'f1-score': 0.9165979438121942, 'support': 9226.0} | {'precision': 0.9016637478108581, 'recall': 0.8528948894226787, 'f1-score': 0.8766015408845188, 'support': 12073.0} | 0.8400 | {'precision': 0.7841948236914985, 'recall': 0.8162084304395008, 'f1-score': 0.7981056979301948, 'support': 27619.0} | {'precision': 0.8489511288560475, 'recall': 0.8400376552373366, 'f1-score': 0.843386093646175, 'support': 27619.0} | |
|
| No log | 6.0 | 246 | 0.4595 | {'precision': 0.5807517554729451, 'recall': 0.6746641074856046, 'f1-score': 0.6241953385127637, 'support': 4168.0} | {'precision': 0.7883110906580764, 'recall': 0.796003717472119, 'f1-score': 0.7921387283236995, 'support': 2152.0} | {'precision': 0.9110802732707088, 'recall': 0.925102969867765, 'f1-score': 0.9180380767989674, 'support': 9226.0} | {'precision': 0.9042363830544677, 'recall': 0.8415472542035948, 'f1-score': 0.8717662705392766, 'support': 12073.0} | 0.8407 | {'precision': 0.7960948756140495, 'recall': 0.8093295122572709, 'f1-score': 0.8015346035436768, 'support': 27619.0} | {'precision': 0.8486726976979458, 'recall': 0.8407255874579094, 'f1-score': 0.8436577064716956, 'support': 27619.0} | |
|
| No log | 7.0 | 287 | 0.5069 | {'precision': 0.6110236220472441, 'recall': 0.5585412667946257, 'f1-score': 0.5836049135121585, 'support': 4168.0} | {'precision': 0.8053691275167785, 'recall': 0.7806691449814126, 'f1-score': 0.7928268050967437, 'support': 2152.0} | {'precision': 0.9251618566882476, 'recall': 0.9138304790808585, 'f1-score': 0.9194612574295218, 'support': 9226.0} | {'precision': 0.8609833465503569, 'recall': 0.8992793837488611, 'f1-score': 0.8797147834541992, 'support': 12073.0} | 0.8435 | {'precision': 0.8006344882006567, 'recall': 0.7880800686514394, 'f1-score': 0.7939019398731558, 'support': 27619.0} | {'precision': 0.8403669956123412, 'recall': 0.8434773163402006, 'f1-score': 0.8415357075120093, 'support': 27619.0} | |
|
| No log | 8.0 | 328 | 0.5486 | {'precision': 0.5794648982391951, 'recall': 0.6079654510556622, 'f1-score': 0.5933731413183467, 'support': 4168.0} | {'precision': 0.7641959254442999, 'recall': 0.8192379182156134, 'f1-score': 0.7907602601480151, 'support': 2152.0} | {'precision': 0.9482497964879637, 'recall': 0.8838066334272707, 'f1-score': 0.9148948106591865, 'support': 9226.0} | {'precision': 0.86709886547812, 'recall': 0.8862751594466992, 'f1-score': 0.8765821488551183, 'support': 12073.0} | 0.8382 | {'precision': 0.7897523714123946, 'recall': 0.7993212905363115, 'f1-score': 0.7939025902451666, 'support': 27619.0} | {'precision': 0.8427820179127555, 'recall': 0.8382273072884608, 'f1-score': 0.8399540584062747, 'support': 27619.0} | |
|
| No log | 9.0 | 369 | 0.5624 | {'precision': 0.5684468999386126, 'recall': 0.6665067178502879, 'f1-score': 0.6135836554389841, 'support': 4168.0} | {'precision': 0.7784669915817457, 'recall': 0.8164498141263941, 'f1-score': 0.7970061238376048, 'support': 2152.0} | {'precision': 0.9420438957475995, 'recall': 0.893236505527856, 'f1-score': 0.9169912095248693, 'support': 9226.0} | {'precision': 0.8849663170461328, 'recall': 0.8596040752091444, 'f1-score': 0.8721008403361344, 'support': 12073.0} | 0.8383 | {'precision': 0.7934810260785227, 'recall': 0.8089492781784205, 'f1-score': 0.7999204572843982, 'support': 27619.0} | {'precision': 0.8479685351639584, 'recall': 0.8383359281653934, 'f1-score': 0.8422320938058151, 'support': 27619.0} | |
|
| No log | 10.0 | 410 | 0.5923 | {'precision': 0.6067892503536068, 'recall': 0.6175623800383877, 'f1-score': 0.612128418549346, 'support': 4168.0} | {'precision': 0.7623089983022071, 'recall': 0.8345724907063197, 'f1-score': 0.7968056787932565, 'support': 2152.0} | {'precision': 0.9368265850062379, 'recall': 0.8952959028831563, 'f1-score': 0.9155905337249902, 'support': 9226.0} | {'precision': 0.8744673877417241, 'recall': 0.8839559347303901, 'f1-score': 0.879186060880669, 'support': 12073.0} | 0.8437 | {'precision': 0.795098055350944, 'recall': 0.8078466770895635, 'f1-score': 0.8009276729870654, 'support': 27619.0} | {'precision': 0.846163633922067, 'recall': 0.8436945580940657, 'f1-score': 0.8446261141401151, 'support': 27619.0} | |
|
| No log | 11.0 | 451 | 0.6036 | {'precision': 0.5938604240282686, 'recall': 0.6451535508637236, 'f1-score': 0.6184452621895125, 'support': 4168.0} | {'precision': 0.7668161434977578, 'recall': 0.7946096654275093, 'f1-score': 0.7804655408489276, 'support': 2152.0} | {'precision': 0.9390562819783969, 'recall': 0.8951875135486668, 'f1-score': 0.9165973031463293, 'support': 9226.0} | {'precision': 0.8781700646444555, 'recall': 0.8776608962146939, 'f1-score': 0.8779154066034218, 'support': 12073.0} | 0.8420 | {'precision': 0.7944757285372197, 'recall': 0.8031529065136485, 'f1-score': 0.7983558781970478, 'support': 27619.0} | {'precision': 0.8469270804932185, 'recall': 0.841956624063145, 'f1-score': 0.8440870820617664, 'support': 27619.0} | |
|
| No log | 12.0 | 492 | 0.6292 | {'precision': 0.594930767425487, 'recall': 0.6082053742802304, 'f1-score': 0.6014948392454621, 'support': 4168.0} | {'precision': 0.7890961262553802, 'recall': 0.766728624535316, 'f1-score': 0.7777515908555267, 'support': 2152.0} | {'precision': 0.9292805354155047, 'recall': 0.9029915456319099, 'f1-score': 0.9159474465394976, 'support': 9226.0} | {'precision': 0.872541050235734, 'recall': 0.8890913608879317, 'f1-score': 0.8807384615384615, 'support': 12073.0} | 0.8418 | {'precision': 0.7964621198330264, 'recall': 0.791754226333847, 'f1-score': 0.7939830845447369, 'support': 27619.0} | {'precision': 0.8430984692266364, 'recall': 0.8418117962272349, 'f1-score': 0.8423345704559697, 'support': 27619.0} | |
|
| 0.2689 | 13.0 | 533 | 0.6506 | {'precision': 0.6016401590457257, 'recall': 0.5808541266794626, 'f1-score': 0.5910644531250001, 'support': 4168.0} | {'precision': 0.7968977217644208, 'recall': 0.7639405204460966, 'f1-score': 0.7800711743772243, 'support': 2152.0} | {'precision': 0.9178990865593737, 'recall': 0.9149143724257534, 'f1-score': 0.9164042992074695, 'support': 9226.0} | {'precision': 0.8670557717250325, 'recall': 0.8859438416300837, 'f1-score': 0.8763980498996273, 'support': 12073.0} | 0.8401 | {'precision': 0.7958731847736381, 'recall': 0.786413215295349, 'f1-score': 0.7909844941523303, 'support': 27619.0} | {'precision': 0.8385191855162285, 'recall': 0.8400738621963141, 'f1-score': 0.8391965505199718, 'support': 27619.0} | |
|
| 0.2689 | 14.0 | 574 | 0.6476 | {'precision': 0.6124620060790273, 'recall': 0.5801343570057581, 'f1-score': 0.5958600295712174, 'support': 4168.0} | {'precision': 0.77728285077951, 'recall': 0.8108736059479554, 'f1-score': 0.7937229929497385, 'support': 2152.0} | {'precision': 0.9248128577719067, 'recall': 0.9105787990461739, 'f1-score': 0.9176406335335883, 'support': 9226.0} | {'precision': 0.8699562469615946, 'recall': 0.8893398492503934, 'f1-score': 0.8795412656154004, 'support': 12073.0} | 0.8437 | {'precision': 0.7961284903980096, 'recall': 0.7977316528125702, 'f1-score': 0.7966912304174861, 'support': 27619.0} | {'precision': 0.8422013661459805, 'recall': 0.8436583511350881, 'f1-score': 0.8427709427870771, 'support': 27619.0} | |
|
| 0.2689 | 15.0 | 615 | 0.6652 | {'precision': 0.5992348158775705, 'recall': 0.6012476007677543, 'f1-score': 0.6002395209580837, 'support': 4168.0} | {'precision': 0.7798372513562387, 'recall': 0.8015799256505576, 'f1-score': 0.7905591200733272, 'support': 2152.0} | {'precision': 0.9388219240391176, 'recall': 0.8948623455451984, 'f1-score': 0.916315205327414, 'support': 9226.0} | {'precision': 0.8667846512750382, 'recall': 0.8924873685082415, 'f1-score': 0.8794482533463925, 'support': 12073.0} | 0.8422 | {'precision': 0.7961696606369912, 'recall': 0.797544310117938, 'f1-score': 0.7966405249263044, 'support': 27619.0} | {'precision': 0.8436975503647769, 'recall': 0.842246279734965, 'f1-score': 0.842701922471951, 'support': 27619.0} | |
|
| 0.2689 | 16.0 | 656 | 0.6609 | {'precision': 0.6078710289236605, 'recall': 0.6151631477927063, 'f1-score': 0.6114953493918436, 'support': 4168.0} | {'precision': 0.782967032967033, 'recall': 0.7946096654275093, 'f1-score': 0.7887453874538746, 'support': 2152.0} | {'precision': 0.934072084172823, 'recall': 0.9045089963147627, 'f1-score': 0.9190528634361235, 'support': 9226.0} | {'precision': 0.8725067166001791, 'recall': 0.8876832601673155, 'f1-score': 0.8800295615043522, 'support': 12073.0} | 0.8449 | {'precision': 0.7993542156659239, 'recall': 0.8004912674255735, 'f1-score': 0.7998307904465485, 'support': 27619.0} | {'precision': 0.8461593157460915, 'recall': 0.8449255946993012, 'f1-score': 0.8454278324403368, 'support': 27619.0} | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.37.2 |
|
- Pytorch 2.2.0+cu121 |
|
- Datasets 2.17.0 |
|
- Tokenizers 0.15.2 |
|
|
