File size: 13,819 Bytes
55a8b23 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
---
license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
- generated_from_trainer
datasets:
- essays_su_g
metrics:
- accuracy
model-index:
- name: longformer-simple
results:
- task:
name: Token Classification
type: token-classification
dataset:
name: essays_su_g
type: essays_su_g
config: simple
split: train[80%:100%]
args: simple
metrics:
- name: Accuracy
type: accuracy
value: 0.8416669683913248
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# longformer-simple
This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co/allenai/longformer-base-4096) on the essays_su_g dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5991
- Claim: {'precision': 0.6002427184466019, 'recall': 0.5933301343570058, 'f1-score': 0.5967664092664092, 'support': 4168.0}
- Majorclaim: {'precision': 0.7753721244925575, 'recall': 0.7987918215613383, 'f1-score': 0.7869077592126345, 'support': 2152.0}
- O: {'precision': 0.9312121891104638, 'recall': 0.9009321482766096, 'f1-score': 0.9158219479947113, 'support': 9226.0}
- Premise: {'precision': 0.8693752023308514, 'recall': 0.8897539965211629, 'f1-score': 0.8794465594170862, 'support': 12073.0}
- Accuracy: 0.8417
- Macro avg: {'precision': 0.7940505585951186, 'recall': 0.7957020251790292, 'f1-score': 0.7947356689727103, 'support': 27619.0}
- Weighted avg: {'precision': 0.8420921444247412, 'recall': 0.8416669683913248, 'f1-score': 0.8417277778228636, 'support': 27619.0}
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 13
### Training results
| Training Loss | Epoch | Step | Validation Loss | Claim | Majorclaim | O | Premise | Accuracy | Macro avg | Weighted avg |
|:-------------:|:-----:|:----:|:---------------:|:--------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|
| No log | 1.0 | 41 | 0.5694 | {'precision': 0.49641025641025643, 'recall': 0.23224568138195778, 'f1-score': 0.3164432821183394, 'support': 4168.0} | {'precision': 0.5319789315274642, 'recall': 0.6570631970260223, 'f1-score': 0.5879417879417879, 'support': 2152.0} | {'precision': 0.9192651679961324, 'recall': 0.8244092781270324, 'f1-score': 0.8692571428571428, 'support': 9226.0} | {'precision': 0.7774988125127231, 'recall': 0.9490598856953533, 'f1-score': 0.8547556881760537, 'support': 12073.0} | 0.7765 | {'precision': 0.6812882921116441, 'recall': 0.6656945105575914, 'f1-score': 0.657099475273331, 'support': 27619.0} | {'precision': 0.7633057030581657, 'recall': 0.776494442231797, 'f1-score': 0.7575733426579333, 'support': 27619.0} |
| No log | 2.0 | 82 | 0.4419 | {'precision': 0.6089438629876308, 'recall': 0.46065259117082535, 'f1-score': 0.5245185084004917, 'support': 4168.0} | {'precision': 0.7108843537414966, 'recall': 0.7769516728624535, 'f1-score': 0.7424511545293072, 'support': 2152.0} | {'precision': 0.9222750963063675, 'recall': 0.8822891827444179, 'f1-score': 0.901839131398183, 'support': 9226.0} | {'precision': 0.8349638771824203, 'recall': 0.9189927938374887, 'f1-score': 0.8749654982059067, 'support': 12073.0} | 0.8265 | {'precision': 0.7692667975544787, 'recall': 0.7597215601537963, 'f1-score': 0.7609435731334722, 'support': 27619.0} | {'precision': 0.820353020671641, 'recall': 0.8264962525797458, 'f1-score': 0.820731174686986, 'support': 27619.0} |
| No log | 3.0 | 123 | 0.4293 | {'precision': 0.559539052496799, 'recall': 0.6290786948176583, 'f1-score': 0.5922746781115881, 'support': 4168.0} | {'precision': 0.7270703472840605, 'recall': 0.7588289962825279, 'f1-score': 0.7426102773988177, 'support': 2152.0} | {'precision': 0.9509253731343283, 'recall': 0.8632126598742683, 'f1-score': 0.9049485824669052, 'support': 9226.0} | {'precision': 0.8691520467836257, 'recall': 0.8863579889008532, 'f1-score': 0.877670699200328, 'support': 12073.0} | 0.8299 | {'precision': 0.7766717049247034, 'recall': 0.7843695849688269, 'f1-score': 0.7793760592944097, 'support': 27619.0} | {'precision': 0.8386735331300186, 'recall': 0.8298634997646548, 'f1-score': 0.8331899108807915, 'support': 27619.0} |
| No log | 4.0 | 164 | 0.4211 | {'precision': 0.6451208032632569, 'recall': 0.4932821497120921, 'f1-score': 0.559075458871516, 'support': 4168.0} | {'precision': 0.78687761749651, 'recall': 0.7857806691449815, 'f1-score': 0.786328760753313, 'support': 2152.0} | {'precision': 0.9139459459459459, 'recall': 0.9163234337741166, 'f1-score': 0.915133145702533, 'support': 9226.0} | {'precision': 0.849535793754316, 'recall': 0.917087716391949, 'f1-score': 0.8820202342069625, 'support': 12073.0} | 0.8426 | {'precision': 0.7988700401150072, 'recall': 0.7781184922557848, 'f1-score': 0.7856393998835811, 'support': 27619.0} | {'precision': 0.8353211584831782, 'recall': 0.8426445562837177, 'f1-score': 0.836889630165822, 'support': 27619.0} |
| No log | 5.0 | 205 | 0.4516 | {'precision': 0.5637651821862348, 'recall': 0.6681861804222649, 'f1-score': 0.6115502854633289, 'support': 4168.0} | {'precision': 0.7106579453636014, 'recall': 0.858271375464684, 'f1-score': 0.7775205219953694, 'support': 2152.0} | {'precision': 0.930641011298803, 'recall': 0.901690873618036, 'f1-score': 0.9159372419488027, 'support': 9226.0} | {'precision': 0.9066511085180864, 'recall': 0.8366603164085149, 'f1-score': 0.8702507107779788, 'support': 12073.0} | 0.8346 | {'precision': 0.7779288118416814, 'recall': 0.8162021864783748, 'f1-score': 0.79381469004637, 'support': 27619.0} | {'precision': 0.8476484297460557, 'recall': 0.8346428183496868, 'f1-score': 0.8392461558560188, 'support': 27619.0} |
| No log | 6.0 | 246 | 0.4529 | {'precision': 0.5875982161817795, 'recall': 0.6638675623800384, 'f1-score': 0.6234088092824153, 'support': 4168.0} | {'precision': 0.7763975155279503, 'recall': 0.8131970260223048, 'f1-score': 0.7943713118474807, 'support': 2152.0} | {'precision': 0.9178008209116439, 'recall': 0.9209841751571646, 'f1-score': 0.9193897424799826, 'support': 9226.0} | {'precision': 0.9001579224425338, 'recall': 0.8498301996189845, 'f1-score': 0.8742703762089388, 'support': 12073.0} | 0.8427 | {'precision': 0.7954886187659769, 'recall': 0.8119697407946231, 'f1-score': 0.8028600599547043, 'support': 27619.0} | {'precision': 0.8492397910801023, 'recall': 0.8426807632426953, 'f1-score': 0.8452590968635983, 'support': 27619.0} |
| No log | 7.0 | 287 | 0.5035 | {'precision': 0.6264514301897479, 'recall': 0.5307101727447217, 'f1-score': 0.574620080529939, 'support': 4168.0} | {'precision': 0.8070770722249152, 'recall': 0.7736988847583643, 'f1-score': 0.7900355871886122, 'support': 2152.0} | {'precision': 0.9218476357267951, 'recall': 0.9128549750704531, 'f1-score': 0.917329266964383, 'support': 9226.0} | {'precision': 0.8542943595313833, 'recall': 0.9120351196885612, 'f1-score': 0.8822209758833427, 'support': 12073.0} | 0.8440 | {'precision': 0.8024176244182103, 'recall': 0.7823247880655251, 'f1-score': 0.7910514776415692, 'support': 27619.0} | {'precision': 0.8387972595060172, 'recall': 0.8439842137658858, 'f1-score': 0.8403456583559027, 'support': 27619.0} |
| No log | 8.0 | 328 | 0.5478 | {'precision': 0.5803487276154571, 'recall': 0.5909309021113244, 'f1-score': 0.5855920114122681, 'support': 4168.0} | {'precision': 0.7547730165464573, 'recall': 0.8266728624535316, 'f1-score': 0.7890884896872921, 'support': 2152.0} | {'precision': 0.948723924950472, 'recall': 0.8823975720789075, 'f1-score': 0.9143595215364744, 'support': 9226.0} | {'precision': 0.8638739245798827, 'recall': 0.8899196554294707, 'f1-score': 0.8767033863729091, 'support': 12073.0} | 0.8374 | {'precision': 0.7869298984230673, 'recall': 0.7974802480183085, 'f1-score': 0.7914358522522359, 'support': 27619.0} | {'precision': 0.8409298617384836, 'recall': 0.8373583402730005, 'f1-score': 0.8385237286921696, 'support': 27619.0} |
| No log | 9.0 | 369 | 0.5551 | {'precision': 0.5595470519328387, 'recall': 0.6876199616122841, 'f1-score': 0.6170075349838535, 'support': 4168.0} | {'precision': 0.7613151152860803, 'recall': 0.8285315985130112, 'f1-score': 0.7935024477080552, 'support': 2152.0} | {'precision': 0.9396424097483203, 'recall': 0.8943203988727509, 'f1-score': 0.916421391681013, 'support': 9226.0} | {'precision': 0.8944082996307368, 'recall': 0.8426240371075955, 'f1-score': 0.8677442743208086, 'support': 12073.0} | 0.8354 | {'precision': 0.788728219149494, 'recall': 0.8132739990264104, 'f1-score': 0.7986689121734325, 'support': 27619.0} | {'precision': 0.84861416106056, 'recall': 0.8354031644882146, 'f1-score': 0.8403810802999596, 'support': 27619.0} |
| No log | 10.0 | 410 | 0.5735 | {'precision': 0.6080889309366131, 'recall': 0.6168426103646834, 'f1-score': 0.6124344926155313, 'support': 4168.0} | {'precision': 0.7508333333333334, 'recall': 0.837360594795539, 'f1-score': 0.7917398945518453, 'support': 2152.0} | {'precision': 0.9385544915640675, 'recall': 0.8923693908519401, 'f1-score': 0.914879431047894, 'support': 9226.0} | {'precision': 0.8762582862754726, 'recall': 0.8868549656257765, 'f1-score': 0.8815247818211758, 'support': 12073.0} | 0.8441 | {'precision': 0.7934337605273717, 'recall': 0.8083568904094847, 'f1-score': 0.8001446500091116, 'support': 27619.0} | {'precision': 0.8468256644647166, 'recall': 0.8440928346428184, 'f1-score': 0.8450623679377253, 'support': 27619.0} |
| No log | 11.0 | 451 | 0.5724 | {'precision': 0.5911991199119911, 'recall': 0.6446737044145874, 'f1-score': 0.6167795248479283, 'support': 4168.0} | {'precision': 0.7570093457943925, 'recall': 0.8280669144981413, 'f1-score': 0.7909454061251665, 'support': 2152.0} | {'precision': 0.9376067402937492, 'recall': 0.8925861695209192, 'f1-score': 0.9145427286356822, 'support': 9226.0} | {'precision': 0.8821311887408897, 'recall': 0.8721941522405368, 'f1-score': 0.8771345272803, 'support': 12073.0} | 0.8412 | {'precision': 0.7919865986852557, 'recall': 0.8093802351685462, 'f1-score': 0.7998505467222692, 'support': 27619.0} | {'precision': 0.8470086415714401, 'recall': 0.8412324848835946, 'f1-score': 0.8436246039246672, 'support': 27619.0} |
| No log | 12.0 | 492 | 0.5952 | {'precision': 0.6041920545941993, 'recall': 0.5947696737044146, 'f1-score': 0.5994438399226211, 'support': 4168.0} | {'precision': 0.7799544419134397, 'recall': 0.7955390334572491, 'f1-score': 0.7876696572348746, 'support': 2152.0} | {'precision': 0.930126130148454, 'recall': 0.9032083243008888, 'f1-score': 0.9164696178168821, 'support': 9226.0} | {'precision': 0.869600388286685, 'recall': 0.8904166321543941, 'f1-score': 0.8798854102721507, 'support': 12073.0} | 0.8427 | {'precision': 0.7959682537356945, 'recall': 0.7959834159042366, 'f1-score': 0.7958671313116321, 'support': 27619.0} | {'precision': 0.8427808250509116, 'recall': 0.8426807632426953, 'f1-score': 0.842599380113732, 'support': 27619.0} |
| 0.2759 | 13.0 | 533 | 0.5991 | {'precision': 0.6002427184466019, 'recall': 0.5933301343570058, 'f1-score': 0.5967664092664092, 'support': 4168.0} | {'precision': 0.7753721244925575, 'recall': 0.7987918215613383, 'f1-score': 0.7869077592126345, 'support': 2152.0} | {'precision': 0.9312121891104638, 'recall': 0.9009321482766096, 'f1-score': 0.9158219479947113, 'support': 9226.0} | {'precision': 0.8693752023308514, 'recall': 0.8897539965211629, 'f1-score': 0.8794465594170862, 'support': 12073.0} | 0.8417 | {'precision': 0.7940505585951186, 'recall': 0.7957020251790292, 'f1-score': 0.7947356689727103, 'support': 27619.0} | {'precision': 0.8420921444247412, 'recall': 0.8416669683913248, 'f1-score': 0.8417277778228636, 'support': 27619.0} |
### Framework versions
- Transformers 4.37.2
- Pytorch 2.2.0+cu121
- Datasets 2.17.0
- Tokenizers 0.15.2
|