meta_data/README_s42_e15.md

metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-simple
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: simple
          split: train[80%:100%]
          args: simple
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8436583511350881

longformer-simple

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

• Loss: 0.6438
• Claim: {'precision': 0.6039084842707341, 'recall': 0.6079654510556622, 'f1-score': 0.6059301769488283, 'support': 4168.0}
• Majorclaim: {'precision': 0.7791218637992832, 'recall': 0.8080855018587361, 'f1-score': 0.7933394160583942, 'support': 2152.0}
• O: {'precision': 0.936604624929498, 'recall': 0.8999566442662043, 'f1-score': 0.9179149853518324, 'support': 9226.0}
• Premise: {'precision': 0.8701119584617881, 'recall': 0.8883458958005467, 'f1-score': 0.8791343907537195, 'support': 12073.0}
• Accuracy: 0.8437
• Macro avg: {'precision': 0.7974367328653259, 'recall': 0.8010883732452874, 'f1-score': 0.7990797422781937, 'support': 27619.0}
• Weighted avg: {'precision': 0.8450608913228282, 'recall': 0.8436583511350881, 'f1-score': 0.8441745376482147, 'support': 27619.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.5691	{'precision': 0.4949392712550607, 'recall': 0.23464491362763915, 'f1-score': 0.318359375, 'support': 4168.0}	{'precision': 0.5329815303430079, 'recall': 0.6570631970260223, 'f1-score': 0.5885535900104059, 'support': 2152.0}	{'precision': 0.919937015503876, 'recall': 0.8232169954476479, 'f1-score': 0.86889371925409, 'support': 9226.0}	{'precision': 0.7775213791231166, 'recall': 0.9488942267870455, 'f1-score': 0.8547021300406611, 'support': 12073.0}	0.7764	{'precision': 0.6813447990562653, 'recall': 0.6659548332220888, 'f1-score': 0.6576272035762892, 'support': 27619.0}	{'precision': 0.7633961277048913, 'recall': 0.7763858213548644, 'f1-score': 0.7577653597350205, 'support': 27619.0}
No log	2.0	82	0.4424	{'precision': 0.6061801446416831, 'recall': 0.44241842610364684, 'f1-score': 0.511511789181692, 'support': 4168.0}	{'precision': 0.6986357999173212, 'recall': 0.7853159851301115, 'f1-score': 0.7394443229052724, 'support': 2152.0}	{'precision': 0.9271515569343904, 'recall': 0.8745935399956645, 'f1-score': 0.900105973562385, 'support': 9226.0}	{'precision': 0.8290598290598291, 'recall': 0.9239625610867225, 'f1-score': 0.8739423378251332, 'support': 12073.0}	0.8240	{'precision': 0.765256832638306, 'recall': 0.7565726280790362, 'f1-score': 0.7562511058686207, 'support': 27619.0}	{'precision': 0.818029713776915, 'recall': 0.8239979724102973, 'f1-score': 0.8175078343477619, 'support': 27619.0}
No log	3.0	123	0.4282	{'precision': 0.5578358208955224, 'recall': 0.6456333973128598, 'f1-score': 0.598532028469751, 'support': 4168.0}	{'precision': 0.7531854648419065, 'recall': 0.741635687732342, 'f1-score': 0.74736595645048, 'support': 2152.0}	{'precision': 0.9484389782403028, 'recall': 0.8692824626056797, 'f1-score': 0.9071372016740188, 'support': 9226.0}	{'precision': 0.872013093289689, 'recall': 0.8826306634639278, 'f1-score': 0.8772897542501955, 'support': 12073.0}	0.8314	{'precision': 0.7828683393168552, 'recall': 0.7847955527787023, 'f1-score': 0.7825812352111113, 'support': 27619.0}	{'precision': 0.8408713896362565, 'recall': 0.831420399000688, 'f1-score': 0.835069338449997, 'support': 27619.0}
No log	4.0	164	0.4201	{'precision': 0.6498756218905473, 'recall': 0.5014395393474088, 'f1-score': 0.566088840736728, 'support': 4168.0}	{'precision': 0.781447963800905, 'recall': 0.8025092936802974, 'f1-score': 0.7918386061439706, 'support': 2152.0}	{'precision': 0.9166757197175448, 'recall': 0.9145892044222849, 'f1-score': 0.9156312733980794, 'support': 9226.0}	{'precision': 0.8523252232830305, 'recall': 0.9169220574836412, 'f1-score': 0.8834443956745541, 'support': 12073.0}	0.8445	{'precision': 0.8000811321730068, 'recall': 0.7838650237334082, 'f1-score': 0.7892507789883332, 'support': 27619.0}	{'precision': 0.8377468489427367, 'recall': 0.8445273181505485, 'f1-score': 0.839166272709442, 'support': 27619.0}
No log	5.0	205	0.4511	{'precision': 0.5708701913186587, 'recall': 0.6657869481765835, 'f1-score': 0.614686011739949, 'support': 4168.0}	{'precision': 0.7117988394584139, 'recall': 0.8550185873605948, 'f1-score': 0.7768629934557737, 'support': 2152.0}	{'precision': 0.9312506998096518, 'recall': 0.9014740949490571, 'f1-score': 0.916120504488627, 'support': 9226.0}	{'precision': 0.9057107276285359, 'recall': 0.8433695021949805, 'f1-score': 0.8734291228822647, 'support': 12073.0}	0.8369	{'precision': 0.7799076145538151, 'recall': 0.816412283170304, 'f1-score': 0.7952746581416535, 'support': 27619.0}	{'precision': 0.8486021445756123, 'recall': 0.8368876498062928, 'f1-score': 0.8411187238429554, 'support': 27619.0}
No log	6.0	246	0.4568	{'precision': 0.5821080969144751, 'recall': 0.6744241842610365, 'f1-score': 0.6248749583194397, 'support': 4168.0}	{'precision': 0.7789237668161435, 'recall': 0.8071561338289963, 'f1-score': 0.7927886809675947, 'support': 2152.0}	{'precision': 0.9134450171821306, 'recall': 0.9219596791675699, 'f1-score': 0.9176825979070017, 'support': 9226.0}	{'precision': 0.9038051209103841, 'recall': 0.8420442309285182, 'f1-score': 0.8718322541915012, 'support': 12073.0}	0.8407	{'precision': 0.7945705004557834, 'recall': 0.8113960570465302, 'f1-score': 0.8017946228463844, 'support': 27619.0}	{'precision': 0.8487473640392945, 'recall': 0.8407255874579094, 'f1-score': 0.8437210080329368, 'support': 27619.0}
No log	7.0	287	0.5084	{'precision': 0.6148536720044174, 'recall': 0.5343090211132437, 'f1-score': 0.5717586649550707, 'support': 4168.0}	{'precision': 0.8070429329474192, 'recall': 0.7774163568773235, 'f1-score': 0.7919526627218936, 'support': 2152.0}	{'precision': 0.9237677984665936, 'recall': 0.914155647084327, 'f1-score': 0.9189365874918283, 'support': 9226.0}	{'precision': 0.8554009692043145, 'recall': 0.9064855462602501, 'f1-score': 0.8802026782482808, 'support': 12073.0}	0.8428	{'precision': 0.8002663431556862, 'recall': 0.7830916428337862, 'f1-score': 0.7907126483542682, 'support': 27619.0}	{'precision': 0.838169524837023, 'recall': 0.8428255910786053, 'f1-score': 0.8397178803143253, 'support': 27619.0}
No log	8.0	328	0.5501	{'precision': 0.5789353438428148, 'recall': 0.6079654510556622, 'f1-score': 0.5930953774136923, 'support': 4168.0}	{'precision': 0.7649107531562909, 'recall': 0.8164498141263941, 'f1-score': 0.7898404135760846, 'support': 2152.0}	{'precision': 0.9487657196087564, 'recall': 0.8831562974203339, 'f1-score': 0.9147861232738297, 'support': 9226.0}	{'precision': 0.8670389253054949, 'recall': 0.8874347718048539, 'f1-score': 0.8771182971756039, 'support': 12073.0}	0.8383	{'precision': 0.7899126854783393, 'recall': 0.7987515836018111, 'f1-score': 0.7937100528598027, 'support': 27619.0}	{'precision': 0.8429039403400853, 'recall': 0.8382997212064158, 'f1-score': 0.8400385270357877, 'support': 27619.0}
No log	9.0	369	0.5615	{'precision': 0.5539168741620379, 'recall': 0.6938579654510557, 'f1-score': 0.6160400468633508, 'support': 4168.0}	{'precision': 0.7715914072775099, 'recall': 0.8178438661710037, 'f1-score': 0.7940446650124069, 'support': 2152.0}	{'precision': 0.9404937990670156, 'recall': 0.8959462388900932, 'f1-score': 0.9176797113516515, 'support': 9226.0}	{'precision': 0.8944209039548022, 'recall': 0.8392280294872857, 'f1-score': 0.865945899747874, 'support': 12073.0}	0.8346	{'precision': 0.7901057461153415, 'recall': 0.8117190249998596, 'f1-score': 0.7984275807438208, 'support': 27619.0}	{'precision': 0.8488551216049527, 'recall': 0.8345704044317318, 'f1-score': 0.8399115427430235, 'support': 27619.0}
No log	10.0	410	0.5889	{'precision': 0.5963951631302761, 'recall': 0.6271593090211133, 'f1-score': 0.6113904806455386, 'support': 4168.0}	{'precision': 0.7538461538461538, 'recall': 0.8424721189591078, 'f1-score': 0.7956989247311828, 'support': 2152.0}	{'precision': 0.9409945004582951, 'recall': 0.8902016041621504, 'f1-score': 0.9148936170212766, 'support': 9226.0}	{'precision': 0.8769726514087416, 'recall': 0.8791518263894641, 'f1-score': 0.8780608868299139, 'support': 12073.0}	0.8420	{'precision': 0.7920521172108667, 'recall': 0.8097462146329589, 'f1-score': 0.800010977306978, 'support': 27619.0}	{'precision': 0.8464230437267781, 'recall': 0.841956624063145, 'f1-score': 0.8437038707660653, 'support': 27619.0}
No log	11.0	451	0.5894	{'precision': 0.5867732872271451, 'recall': 0.6513915547024952, 'f1-score': 0.6173962478681069, 'support': 4168.0}	{'precision': 0.7666963490650045, 'recall': 0.800185873605948, 'f1-score': 0.7830832196452934, 'support': 2152.0}	{'precision': 0.9389980688401681, 'recall': 0.8959462388900932, 'f1-score': 0.9169671085473403, 'support': 9226.0}	{'precision': 0.8817717491417567, 'recall': 0.8722769816946906, 'f1-score': 0.8769986675549633, 'support': 12073.0}	0.8412	{'precision': 0.7935598635685186, 'recall': 0.8049501622233067, 'f1-score': 0.798611310903926, 'support': 27619.0}	{'precision': 0.8474031686468898, 'recall': 0.8412324848835946, 'f1-score': 0.8438555380947815, 'support': 27619.0}
No log	12.0	492	0.6198	{'precision': 0.5958633511503603, 'recall': 0.6151631477927063, 'f1-score': 0.6053594616928344, 'support': 4168.0}	{'precision': 0.7789770061004223, 'recall': 0.7713754646840149, 'f1-score': 0.7751575998132151, 'support': 2152.0}	{'precision': 0.9328919313208394, 'recall': 0.901040537611099, 'f1-score': 0.9166896399625074, 'support': 9226.0}	{'precision': 0.8742056379338439, 'recall': 0.8887600430713162, 'f1-score': 0.8814227625580152, 'support': 12073.0}	0.8424	{'precision': 0.7954844816263664, 'recall': 0.7940847982897842, 'f1-score': 0.7946573660066429, 'support': 27619.0}	{'precision': 0.8443847565032829, 'recall': 0.8424273145298526, 'f1-score': 0.8432627184833189, 'support': 27619.0}
0.271	13.0	533	0.6308	{'precision': 0.5984138428262437, 'recall': 0.5974088291746641, 'f1-score': 0.597910913675111, 'support': 4168.0}	{'precision': 0.7893231649189705, 'recall': 0.7695167286245354, 'f1-score': 0.779294117647059, 'support': 2152.0}	{'precision': 0.9218476357267951, 'recall': 0.9128549750704531, 'f1-score': 0.917329266964383, 'support': 9226.0}	{'precision': 0.8714005235602095, 'recall': 0.8822993456473122, 'f1-score': 0.8768160678273037, 'support': 12073.0}	0.8407	{'precision': 0.7952462917580546, 'recall': 0.7905199696292412, 'f1-score': 0.7928375915284641, 'support': 27619.0}	{'precision': 0.8406603119578272, 'recall': 0.8407255874579094, 'f1-score': 0.8406609157922722, 'support': 27619.0}
0.271	14.0	574	0.6361	{'precision': 0.6123370110330993, 'recall': 0.5858925143953935, 'f1-score': 0.5988229524276607, 'support': 4168.0}	{'precision': 0.7828622700762674, 'recall': 0.8108736059479554, 'f1-score': 0.7966217758502625, 'support': 2152.0}	{'precision': 0.9273249392533687, 'recall': 0.9100368523737264, 'f1-score': 0.9185995623632386, 'support': 9226.0}	{'precision': 0.8696145124716553, 'recall': 0.8894226787045474, 'f1-score': 0.879407067687646, 'support': 12073.0}	0.8444	{'precision': 0.7980346832085977, 'recall': 0.7990564128554057, 'f1-score': 0.798362839582202, 'support': 27619.0}	{'precision': 0.8433070048087172, 'recall': 0.8443824903146385, 'f1-score': 0.8437056091062111, 'support': 27619.0}
0.271	15.0	615	0.6438	{'precision': 0.6039084842707341, 'recall': 0.6079654510556622, 'f1-score': 0.6059301769488283, 'support': 4168.0}	{'precision': 0.7791218637992832, 'recall': 0.8080855018587361, 'f1-score': 0.7933394160583942, 'support': 2152.0}	{'precision': 0.936604624929498, 'recall': 0.8999566442662043, 'f1-score': 0.9179149853518324, 'support': 9226.0}	{'precision': 0.8701119584617881, 'recall': 0.8883458958005467, 'f1-score': 0.8791343907537195, 'support': 12073.0}	0.8437	{'precision': 0.7974367328653259, 'recall': 0.8010883732452874, 'f1-score': 0.7990797422781937, 'support': 27619.0}	{'precision': 0.8450608913228282, 'recall': 0.8436583511350881, 'f1-score': 0.8441745376482147, 'support': 27619.0}

Framework versions

• Transformers 4.37.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.0
• Tokenizers 0.15.2