File size: 15,322 Bytes
55a8b23 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
---
license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
- generated_from_trainer
datasets:
- essays_su_g
metrics:
- accuracy
model-index:
- name: longformer-simple
results:
- task:
name: Token Classification
type: token-classification
dataset:
name: essays_su_g
type: essays_su_g
config: simple
split: train[80%:100%]
args: simple
metrics:
- name: Accuracy
type: accuracy
value: 0.8436583511350881
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# longformer-simple
This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co/allenai/longformer-base-4096) on the essays_su_g dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6438
- Claim: {'precision': 0.6039084842707341, 'recall': 0.6079654510556622, 'f1-score': 0.6059301769488283, 'support': 4168.0}
- Majorclaim: {'precision': 0.7791218637992832, 'recall': 0.8080855018587361, 'f1-score': 0.7933394160583942, 'support': 2152.0}
- O: {'precision': 0.936604624929498, 'recall': 0.8999566442662043, 'f1-score': 0.9179149853518324, 'support': 9226.0}
- Premise: {'precision': 0.8701119584617881, 'recall': 0.8883458958005467, 'f1-score': 0.8791343907537195, 'support': 12073.0}
- Accuracy: 0.8437
- Macro avg: {'precision': 0.7974367328653259, 'recall': 0.8010883732452874, 'f1-score': 0.7990797422781937, 'support': 27619.0}
- Weighted avg: {'precision': 0.8450608913228282, 'recall': 0.8436583511350881, 'f1-score': 0.8441745376482147, 'support': 27619.0}
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 15
### Training results
| Training Loss | Epoch | Step | Validation Loss | Claim | Majorclaim | O | Premise | Accuracy | Macro avg | Weighted avg |
|:-------------:|:-----:|:----:|:---------------:|:------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|
| No log | 1.0 | 41 | 0.5691 | {'precision': 0.4949392712550607, 'recall': 0.23464491362763915, 'f1-score': 0.318359375, 'support': 4168.0} | {'precision': 0.5329815303430079, 'recall': 0.6570631970260223, 'f1-score': 0.5885535900104059, 'support': 2152.0} | {'precision': 0.919937015503876, 'recall': 0.8232169954476479, 'f1-score': 0.86889371925409, 'support': 9226.0} | {'precision': 0.7775213791231166, 'recall': 0.9488942267870455, 'f1-score': 0.8547021300406611, 'support': 12073.0} | 0.7764 | {'precision': 0.6813447990562653, 'recall': 0.6659548332220888, 'f1-score': 0.6576272035762892, 'support': 27619.0} | {'precision': 0.7633961277048913, 'recall': 0.7763858213548644, 'f1-score': 0.7577653597350205, 'support': 27619.0} |
| No log | 2.0 | 82 | 0.4424 | {'precision': 0.6061801446416831, 'recall': 0.44241842610364684, 'f1-score': 0.511511789181692, 'support': 4168.0} | {'precision': 0.6986357999173212, 'recall': 0.7853159851301115, 'f1-score': 0.7394443229052724, 'support': 2152.0} | {'precision': 0.9271515569343904, 'recall': 0.8745935399956645, 'f1-score': 0.900105973562385, 'support': 9226.0} | {'precision': 0.8290598290598291, 'recall': 0.9239625610867225, 'f1-score': 0.8739423378251332, 'support': 12073.0} | 0.8240 | {'precision': 0.765256832638306, 'recall': 0.7565726280790362, 'f1-score': 0.7562511058686207, 'support': 27619.0} | {'precision': 0.818029713776915, 'recall': 0.8239979724102973, 'f1-score': 0.8175078343477619, 'support': 27619.0} |
| No log | 3.0 | 123 | 0.4282 | {'precision': 0.5578358208955224, 'recall': 0.6456333973128598, 'f1-score': 0.598532028469751, 'support': 4168.0} | {'precision': 0.7531854648419065, 'recall': 0.741635687732342, 'f1-score': 0.74736595645048, 'support': 2152.0} | {'precision': 0.9484389782403028, 'recall': 0.8692824626056797, 'f1-score': 0.9071372016740188, 'support': 9226.0} | {'precision': 0.872013093289689, 'recall': 0.8826306634639278, 'f1-score': 0.8772897542501955, 'support': 12073.0} | 0.8314 | {'precision': 0.7828683393168552, 'recall': 0.7847955527787023, 'f1-score': 0.7825812352111113, 'support': 27619.0} | {'precision': 0.8408713896362565, 'recall': 0.831420399000688, 'f1-score': 0.835069338449997, 'support': 27619.0} |
| No log | 4.0 | 164 | 0.4201 | {'precision': 0.6498756218905473, 'recall': 0.5014395393474088, 'f1-score': 0.566088840736728, 'support': 4168.0} | {'precision': 0.781447963800905, 'recall': 0.8025092936802974, 'f1-score': 0.7918386061439706, 'support': 2152.0} | {'precision': 0.9166757197175448, 'recall': 0.9145892044222849, 'f1-score': 0.9156312733980794, 'support': 9226.0} | {'precision': 0.8523252232830305, 'recall': 0.9169220574836412, 'f1-score': 0.8834443956745541, 'support': 12073.0} | 0.8445 | {'precision': 0.8000811321730068, 'recall': 0.7838650237334082, 'f1-score': 0.7892507789883332, 'support': 27619.0} | {'precision': 0.8377468489427367, 'recall': 0.8445273181505485, 'f1-score': 0.839166272709442, 'support': 27619.0} |
| No log | 5.0 | 205 | 0.4511 | {'precision': 0.5708701913186587, 'recall': 0.6657869481765835, 'f1-score': 0.614686011739949, 'support': 4168.0} | {'precision': 0.7117988394584139, 'recall': 0.8550185873605948, 'f1-score': 0.7768629934557737, 'support': 2152.0} | {'precision': 0.9312506998096518, 'recall': 0.9014740949490571, 'f1-score': 0.916120504488627, 'support': 9226.0} | {'precision': 0.9057107276285359, 'recall': 0.8433695021949805, 'f1-score': 0.8734291228822647, 'support': 12073.0} | 0.8369 | {'precision': 0.7799076145538151, 'recall': 0.816412283170304, 'f1-score': 0.7952746581416535, 'support': 27619.0} | {'precision': 0.8486021445756123, 'recall': 0.8368876498062928, 'f1-score': 0.8411187238429554, 'support': 27619.0} |
| No log | 6.0 | 246 | 0.4568 | {'precision': 0.5821080969144751, 'recall': 0.6744241842610365, 'f1-score': 0.6248749583194397, 'support': 4168.0} | {'precision': 0.7789237668161435, 'recall': 0.8071561338289963, 'f1-score': 0.7927886809675947, 'support': 2152.0} | {'precision': 0.9134450171821306, 'recall': 0.9219596791675699, 'f1-score': 0.9176825979070017, 'support': 9226.0} | {'precision': 0.9038051209103841, 'recall': 0.8420442309285182, 'f1-score': 0.8718322541915012, 'support': 12073.0} | 0.8407 | {'precision': 0.7945705004557834, 'recall': 0.8113960570465302, 'f1-score': 0.8017946228463844, 'support': 27619.0} | {'precision': 0.8487473640392945, 'recall': 0.8407255874579094, 'f1-score': 0.8437210080329368, 'support': 27619.0} |
| No log | 7.0 | 287 | 0.5084 | {'precision': 0.6148536720044174, 'recall': 0.5343090211132437, 'f1-score': 0.5717586649550707, 'support': 4168.0} | {'precision': 0.8070429329474192, 'recall': 0.7774163568773235, 'f1-score': 0.7919526627218936, 'support': 2152.0} | {'precision': 0.9237677984665936, 'recall': 0.914155647084327, 'f1-score': 0.9189365874918283, 'support': 9226.0} | {'precision': 0.8554009692043145, 'recall': 0.9064855462602501, 'f1-score': 0.8802026782482808, 'support': 12073.0} | 0.8428 | {'precision': 0.8002663431556862, 'recall': 0.7830916428337862, 'f1-score': 0.7907126483542682, 'support': 27619.0} | {'precision': 0.838169524837023, 'recall': 0.8428255910786053, 'f1-score': 0.8397178803143253, 'support': 27619.0} |
| No log | 8.0 | 328 | 0.5501 | {'precision': 0.5789353438428148, 'recall': 0.6079654510556622, 'f1-score': 0.5930953774136923, 'support': 4168.0} | {'precision': 0.7649107531562909, 'recall': 0.8164498141263941, 'f1-score': 0.7898404135760846, 'support': 2152.0} | {'precision': 0.9487657196087564, 'recall': 0.8831562974203339, 'f1-score': 0.9147861232738297, 'support': 9226.0} | {'precision': 0.8670389253054949, 'recall': 0.8874347718048539, 'f1-score': 0.8771182971756039, 'support': 12073.0} | 0.8383 | {'precision': 0.7899126854783393, 'recall': 0.7987515836018111, 'f1-score': 0.7937100528598027, 'support': 27619.0} | {'precision': 0.8429039403400853, 'recall': 0.8382997212064158, 'f1-score': 0.8400385270357877, 'support': 27619.0} |
| No log | 9.0 | 369 | 0.5615 | {'precision': 0.5539168741620379, 'recall': 0.6938579654510557, 'f1-score': 0.6160400468633508, 'support': 4168.0} | {'precision': 0.7715914072775099, 'recall': 0.8178438661710037, 'f1-score': 0.7940446650124069, 'support': 2152.0} | {'precision': 0.9404937990670156, 'recall': 0.8959462388900932, 'f1-score': 0.9176797113516515, 'support': 9226.0} | {'precision': 0.8944209039548022, 'recall': 0.8392280294872857, 'f1-score': 0.865945899747874, 'support': 12073.0} | 0.8346 | {'precision': 0.7901057461153415, 'recall': 0.8117190249998596, 'f1-score': 0.7984275807438208, 'support': 27619.0} | {'precision': 0.8488551216049527, 'recall': 0.8345704044317318, 'f1-score': 0.8399115427430235, 'support': 27619.0} |
| No log | 10.0 | 410 | 0.5889 | {'precision': 0.5963951631302761, 'recall': 0.6271593090211133, 'f1-score': 0.6113904806455386, 'support': 4168.0} | {'precision': 0.7538461538461538, 'recall': 0.8424721189591078, 'f1-score': 0.7956989247311828, 'support': 2152.0} | {'precision': 0.9409945004582951, 'recall': 0.8902016041621504, 'f1-score': 0.9148936170212766, 'support': 9226.0} | {'precision': 0.8769726514087416, 'recall': 0.8791518263894641, 'f1-score': 0.8780608868299139, 'support': 12073.0} | 0.8420 | {'precision': 0.7920521172108667, 'recall': 0.8097462146329589, 'f1-score': 0.800010977306978, 'support': 27619.0} | {'precision': 0.8464230437267781, 'recall': 0.841956624063145, 'f1-score': 0.8437038707660653, 'support': 27619.0} |
| No log | 11.0 | 451 | 0.5894 | {'precision': 0.5867732872271451, 'recall': 0.6513915547024952, 'f1-score': 0.6173962478681069, 'support': 4168.0} | {'precision': 0.7666963490650045, 'recall': 0.800185873605948, 'f1-score': 0.7830832196452934, 'support': 2152.0} | {'precision': 0.9389980688401681, 'recall': 0.8959462388900932, 'f1-score': 0.9169671085473403, 'support': 9226.0} | {'precision': 0.8817717491417567, 'recall': 0.8722769816946906, 'f1-score': 0.8769986675549633, 'support': 12073.0} | 0.8412 | {'precision': 0.7935598635685186, 'recall': 0.8049501622233067, 'f1-score': 0.798611310903926, 'support': 27619.0} | {'precision': 0.8474031686468898, 'recall': 0.8412324848835946, 'f1-score': 0.8438555380947815, 'support': 27619.0} |
| No log | 12.0 | 492 | 0.6198 | {'precision': 0.5958633511503603, 'recall': 0.6151631477927063, 'f1-score': 0.6053594616928344, 'support': 4168.0} | {'precision': 0.7789770061004223, 'recall': 0.7713754646840149, 'f1-score': 0.7751575998132151, 'support': 2152.0} | {'precision': 0.9328919313208394, 'recall': 0.901040537611099, 'f1-score': 0.9166896399625074, 'support': 9226.0} | {'precision': 0.8742056379338439, 'recall': 0.8887600430713162, 'f1-score': 0.8814227625580152, 'support': 12073.0} | 0.8424 | {'precision': 0.7954844816263664, 'recall': 0.7940847982897842, 'f1-score': 0.7946573660066429, 'support': 27619.0} | {'precision': 0.8443847565032829, 'recall': 0.8424273145298526, 'f1-score': 0.8432627184833189, 'support': 27619.0} |
| 0.271 | 13.0 | 533 | 0.6308 | {'precision': 0.5984138428262437, 'recall': 0.5974088291746641, 'f1-score': 0.597910913675111, 'support': 4168.0} | {'precision': 0.7893231649189705, 'recall': 0.7695167286245354, 'f1-score': 0.779294117647059, 'support': 2152.0} | {'precision': 0.9218476357267951, 'recall': 0.9128549750704531, 'f1-score': 0.917329266964383, 'support': 9226.0} | {'precision': 0.8714005235602095, 'recall': 0.8822993456473122, 'f1-score': 0.8768160678273037, 'support': 12073.0} | 0.8407 | {'precision': 0.7952462917580546, 'recall': 0.7905199696292412, 'f1-score': 0.7928375915284641, 'support': 27619.0} | {'precision': 0.8406603119578272, 'recall': 0.8407255874579094, 'f1-score': 0.8406609157922722, 'support': 27619.0} |
| 0.271 | 14.0 | 574 | 0.6361 | {'precision': 0.6123370110330993, 'recall': 0.5858925143953935, 'f1-score': 0.5988229524276607, 'support': 4168.0} | {'precision': 0.7828622700762674, 'recall': 0.8108736059479554, 'f1-score': 0.7966217758502625, 'support': 2152.0} | {'precision': 0.9273249392533687, 'recall': 0.9100368523737264, 'f1-score': 0.9185995623632386, 'support': 9226.0} | {'precision': 0.8696145124716553, 'recall': 0.8894226787045474, 'f1-score': 0.879407067687646, 'support': 12073.0} | 0.8444 | {'precision': 0.7980346832085977, 'recall': 0.7990564128554057, 'f1-score': 0.798362839582202, 'support': 27619.0} | {'precision': 0.8433070048087172, 'recall': 0.8443824903146385, 'f1-score': 0.8437056091062111, 'support': 27619.0} |
| 0.271 | 15.0 | 615 | 0.6438 | {'precision': 0.6039084842707341, 'recall': 0.6079654510556622, 'f1-score': 0.6059301769488283, 'support': 4168.0} | {'precision': 0.7791218637992832, 'recall': 0.8080855018587361, 'f1-score': 0.7933394160583942, 'support': 2152.0} | {'precision': 0.936604624929498, 'recall': 0.8999566442662043, 'f1-score': 0.9179149853518324, 'support': 9226.0} | {'precision': 0.8701119584617881, 'recall': 0.8883458958005467, 'f1-score': 0.8791343907537195, 'support': 12073.0} | 0.8437 | {'precision': 0.7974367328653259, 'recall': 0.8010883732452874, 'f1-score': 0.7990797422781937, 'support': 27619.0} | {'precision': 0.8450608913228282, 'recall': 0.8436583511350881, 'f1-score': 0.8441745376482147, 'support': 27619.0} |
### Framework versions
- Transformers 4.37.2
- Pytorch 2.2.0+cu121
- Datasets 2.17.0
- Tokenizers 0.15.2
|