--- license: apache-2.0 base_model: allenai/longformer-base-4096 tags: - generated_from_trainer datasets: - essays_su_g metrics: - accuracy model-index: - name: longformer-simple results: - task: name: Token Classification type: token-classification dataset: name: essays_su_g type: essays_su_g config: simple split: train[80%:100%] args: simple metrics: - name: Accuracy type: accuracy value: 0.8439117998479307 --- # longformer-simple This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co/allenai/longformer-base-4096) on the essays_su_g dataset. It achieves the following results on the evaluation set: - Loss: 0.5465 - Claim: {'precision': 0.6066587395957194, 'recall': 0.6120441458733206, 'f1-score': 0.609339543771647, 'support': 4168.0} - Majorclaim: {'precision': 0.7760070827799912, 'recall': 0.8145910780669146, 'f1-score': 0.7948311040580367, 'support': 2152.0} - O: {'precision': 0.9332207207207207, 'recall': 0.8982224149143724, 'f1-score': 0.9153871644758642, 'support': 9226.0} - Premise: {'precision': 0.8730753564154786, 'recall': 0.8876832601673155, 'f1-score': 0.8803187120091999, 'support': 12073.0} - Accuracy: 0.8439 - Macro avg: {'precision': 0.7972404748779774, 'recall': 0.8031352247554808, 'f1-score': 0.799969131078687, 'support': 27619.0} - Weighted avg: {'precision': 0.8453982409265701, 'recall': 0.8439117998479307, 'f1-score': 0.8444785670702963, 'support': 27619.0} ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 11 ### Training results | Training Loss | Epoch | Step | Validation Loss | Claim | Majorclaim | O | Premise | Accuracy | Macro avg | Weighted avg | |:-------------:|:-----:|:----:|:---------------:|:---------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:| | No log | 1.0 | 41 | 0.5694 | {'precision': 0.49896907216494846, 'recall': 0.23224568138195778, 'f1-score': 0.31696136214800263, 'support': 4168.0} | {'precision': 0.5306275836151823, 'recall': 0.6561338289962825, 'f1-score': 0.5867442343652607, 'support': 2152.0} | {'precision': 0.9175605640592985, 'recall': 0.8251680034684588, 'f1-score': 0.8689151401015808, 'support': 9226.0} | {'precision': 0.7781400720059779, 'recall': 0.9488113973328915, 'f1-score': 0.8550421736209599, 'support': 12073.0} | 0.7766 | {'precision': 0.6813243229613518, 'recall': 0.6655897277948977, 'f1-score': 0.656915727558951, 'support': 27619.0} | {'precision': 0.7632974584909895, 'recall': 0.776566856149752, 'f1-score': 0.7575692021611915, 'support': 27619.0} | | No log | 2.0 | 82 | 0.4421 | {'precision': 0.6022099447513812, 'recall': 0.47072936660268716, 'f1-score': 0.5284136816590359, 'support': 4168.0} | {'precision': 0.7127335940895263, 'recall': 0.7620817843866171, 'f1-score': 0.7365820794969683, 'support': 2152.0} | {'precision': 0.9157366071428571, 'recall': 0.8893344894862345, 'f1-score': 0.902342461233916, 'support': 9226.0} | {'precision': 0.8403053435114504, 'recall': 0.9117866313260996, 'f1-score': 0.8745878520637191, 'support': 12073.0} | 0.8261 | {'precision': 0.7677463723738037, 'recall': 0.7584830679504097, 'f1-score': 0.7604815186134098, 'support': 27619.0} | {'precision': 0.8196316337998537, 'recall': 0.8260617690720157, 'f1-score': 0.820864750553667, 'support': 27619.0} | | No log | 3.0 | 123 | 0.4321 | {'precision': 0.5622799295774648, 'recall': 0.613003838771593, 'f1-score': 0.5865472910927456, 'support': 4168.0} | {'precision': 0.7040472175379426, 'recall': 0.7760223048327137, 'f1-score': 0.7382847038019452, 'support': 2152.0} | {'precision': 0.9531174480425326, 'recall': 0.8549750704530674, 'f1-score': 0.9013826991201006, 'support': 9226.0} | {'precision': 0.865615192725517, 'recall': 0.8909964383334714, 'f1-score': 0.8781224489795918, 'support': 12073.0} | 0.8281 | {'precision': 0.7712649469708642, 'recall': 0.7837494130977114, 'f1-score': 0.7760842857485957, 'support': 27619.0} | {'precision': 0.836479458200373, 'recall': 0.828053151815779, 'f1-score': 0.8309948550081105, 'support': 27619.0} | | No log | 4.0 | 164 | 0.4226 | {'precision': 0.6349646044936904, 'recall': 0.4949616122840691, 'f1-score': 0.5562896049615748, 'support': 4168.0} | {'precision': 0.7841284837033538, 'recall': 0.7713754646840149, 'f1-score': 0.777699695479035, 'support': 2152.0} | {'precision': 0.9162947643409165, 'recall': 0.9124214177324951, 'f1-score': 0.91435398902949, 'support': 9226.0} | {'precision': 0.8469309658656053, 'recall': 0.9165907396670256, 'f1-score': 0.8803850590715622, 'support': 12073.0} | 0.8403 | {'precision': 0.7955797046008914, 'recall': 0.7738373085919011, 'f1-score': 0.7821820871354155, 'support': 27619.0} | {'precision': 0.833220247480505, 'recall': 0.8402548969912017, 'f1-score': 0.8348218088673657, 'support': 27619.0} | | No log | 5.0 | 205 | 0.4429 | {'precision': 0.5707901685963843, 'recall': 0.6741842610364683, 'f1-score': 0.6181938180618194, 'support': 4168.0} | {'precision': 0.7293650793650793, 'recall': 0.854089219330855, 'f1-score': 0.7868150684931506, 'support': 2152.0} | {'precision': 0.9286749136297783, 'recall': 0.9032083243008888, 'f1-score': 0.9157646024506841, 'support': 9226.0} | {'precision': 0.9048469160046416, 'recall': 0.8396421767580552, 'f1-score': 0.871025949475855, 'support': 12073.0} | 0.8370 | {'precision': 0.7834192693989709, 'recall': 0.8177809953565668, 'f1-score': 0.7979498596203773, 'support': 27619.0} | {'precision': 0.8487207590273272, 'recall': 0.8370324776422028, 'f1-score': 0.8412541500891029, 'support': 27619.0} | | No log | 6.0 | 246 | 0.4469 | {'precision': 0.6041713388652165, 'recall': 0.6463531669865643, 'f1-score': 0.6245508287933232, 'support': 4168.0} | {'precision': 0.7802893309222423, 'recall': 0.8020446096654275, 'f1-score': 0.7910174152153987, 'support': 2152.0} | {'precision': 0.9200608232866297, 'recall': 0.9181660524604379, 'f1-score': 0.9191124613464982, 'support': 9226.0} | {'precision': 0.8927689293927263, 'recall': 0.8682183384411497, 'f1-score': 0.8803224993701184, 'support': 12073.0} | 0.8463 | {'precision': 0.7993226056167038, 'recall': 0.8086955418883948, 'f1-score': 0.8037508011813346, 'support': 27619.0} | {'precision': 0.8495691089733778, 'recall': 0.8462652521814693, 'f1-score': 0.8477230325222614, 'support': 27619.0} | | No log | 7.0 | 287 | 0.4855 | {'precision': 0.6054081121682524, 'recall': 0.5801343570057581, 'f1-score': 0.5925018377848567, 'support': 4168.0} | {'precision': 0.7954866008462623, 'recall': 0.7862453531598513, 'f1-score': 0.7908389810703436, 'support': 2152.0} | {'precision': 0.9300268696820421, 'recall': 0.9003902016041622, 'f1-score': 0.9149686088776298, 'support': 9226.0} | {'precision': 0.8639185102657966, 'recall': 0.8991965542947072, 'f1-score': 0.8812045943423028, 'support': 12073.0} | 0.8426 | {'precision': 0.7987100232405884, 'recall': 0.7914916165161198, 'f1-score': 0.7948785055187831, 'support': 27619.0} | {'precision': 0.8416577084856046, 'recall': 0.8426445562837177, 'f1-score': 0.8418739490984576, 'support': 27619.0} | | No log | 8.0 | 328 | 0.5255 | {'precision': 0.5770187740330242, 'recall': 0.6120441458733206, 'f1-score': 0.5940156013505647, 'support': 4168.0} | {'precision': 0.7475521498510004, 'recall': 0.8159851301115242, 'f1-score': 0.7802710508775829, 'support': 2152.0} | {'precision': 0.9392967586759822, 'recall': 0.8889009321482766, 'f1-score': 0.9134042434705129, 'support': 9226.0} | {'precision': 0.8744017164548605, 'recall': 0.8776608962146939, 'f1-score': 0.8760282749782976, 'support': 12073.0} | 0.8365 | {'precision': 0.7845673497537168, 'recall': 0.7986477760869539, 'f1-score': 0.7909297926692396, 'support': 27619.0} | {'precision': 0.841317581916548, 'recall': 0.8365255802165176, 'f1-score': 0.8384936906473678, 'support': 27619.0} | | No log | 9.0 | 369 | 0.5319 | {'precision': 0.5885775862068966, 'recall': 0.6552303262955854, 'f1-score': 0.6201180744777476, 'support': 4168.0} | {'precision': 0.7612709317303564, 'recall': 0.8238847583643123, 'f1-score': 0.7913412184780183, 'support': 2152.0} | {'precision': 0.9386587319752804, 'recall': 0.8890093214827661, 'f1-score': 0.9131596526386107, 'support': 9226.0} | {'precision': 0.8836467427803896, 'recall': 0.8718628344239211, 'f1-score': 0.8777152386908486, 'support': 12073.0} | 0.8412 | {'precision': 0.7930384981732308, 'recall': 0.8099968101416463, 'f1-score': 0.8005835460713063, 'support': 27619.0} | {'precision': 0.8479589779204769, 'recall': 0.8411600709656396, 'f1-score': 0.8439511013630612, 'support': 27619.0} | | No log | 10.0 | 410 | 0.5409 | {'precision': 0.6034322820037106, 'recall': 0.6242802303262955, 'f1-score': 0.6136792452830189, 'support': 4168.0} | {'precision': 0.7689295039164491, 'recall': 0.8210966542750929, 'f1-score': 0.7941573033707865, 'support': 2152.0} | {'precision': 0.9326761510025765, 'recall': 0.9024495989594624, 'f1-score': 0.9173139425990195, 'support': 9226.0} | {'precision': 0.8804833636815097, 'recall': 0.8811397332891576, 'f1-score': 0.8808114262057546, 'support': 12073.0} | 0.8448 | {'precision': 0.7963803251510615, 'recall': 0.8072415542125021, 'f1-score': 0.8014904793646449, 'support': 27619.0} | {'precision': 0.8474161940220971, 'recall': 0.8448169738223686, 'f1-score': 0.8459399831345878, 'support': 27619.0} | | No log | 11.0 | 451 | 0.5465 | {'precision': 0.6066587395957194, 'recall': 0.6120441458733206, 'f1-score': 0.609339543771647, 'support': 4168.0} | {'precision': 0.7760070827799912, 'recall': 0.8145910780669146, 'f1-score': 0.7948311040580367, 'support': 2152.0} | {'precision': 0.9332207207207207, 'recall': 0.8982224149143724, 'f1-score': 0.9153871644758642, 'support': 9226.0} | {'precision': 0.8730753564154786, 'recall': 0.8876832601673155, 'f1-score': 0.8803187120091999, 'support': 12073.0} | 0.8439 | {'precision': 0.7972404748779774, 'recall': 0.8031352247554808, 'f1-score': 0.799969131078687, 'support': 27619.0} | {'precision': 0.8453982409265701, 'recall': 0.8439117998479307, 'f1-score': 0.8444785670702963, 'support': 27619.0} | ### Framework versions - Transformers 4.37.2 - Pytorch 2.2.0+cu121 - Datasets 2.17.0 - Tokenizers 0.15.2