meta_data/README_s42_e13.md

metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-simple
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: simple
          split: train[80%:100%]
          args: simple
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8416669683913248

longformer-simple

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

• Loss: 0.5991
• Claim: {'precision': 0.6002427184466019, 'recall': 0.5933301343570058, 'f1-score': 0.5967664092664092, 'support': 4168.0}
• Majorclaim: {'precision': 0.7753721244925575, 'recall': 0.7987918215613383, 'f1-score': 0.7869077592126345, 'support': 2152.0}
• O: {'precision': 0.9312121891104638, 'recall': 0.9009321482766096, 'f1-score': 0.9158219479947113, 'support': 9226.0}
• Premise: {'precision': 0.8693752023308514, 'recall': 0.8897539965211629, 'f1-score': 0.8794465594170862, 'support': 12073.0}
• Accuracy: 0.8417
• Macro avg: {'precision': 0.7940505585951186, 'recall': 0.7957020251790292, 'f1-score': 0.7947356689727103, 'support': 27619.0}
• Weighted avg: {'precision': 0.8420921444247412, 'recall': 0.8416669683913248, 'f1-score': 0.8417277778228636, 'support': 27619.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 13

Training results

Training Loss	Epoch	Step	Validation Loss	Claim	Majorclaim	O	Premise	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.5694	{'precision': 0.49641025641025643, 'recall': 0.23224568138195778, 'f1-score': 0.3164432821183394, 'support': 4168.0}	{'precision': 0.5319789315274642, 'recall': 0.6570631970260223, 'f1-score': 0.5879417879417879, 'support': 2152.0}	{'precision': 0.9192651679961324, 'recall': 0.8244092781270324, 'f1-score': 0.8692571428571428, 'support': 9226.0}	{'precision': 0.7774988125127231, 'recall': 0.9490598856953533, 'f1-score': 0.8547556881760537, 'support': 12073.0}	0.7765	{'precision': 0.6812882921116441, 'recall': 0.6656945105575914, 'f1-score': 0.657099475273331, 'support': 27619.0}	{'precision': 0.7633057030581657, 'recall': 0.776494442231797, 'f1-score': 0.7575733426579333, 'support': 27619.0}
No log	2.0	82	0.4419	{'precision': 0.6089438629876308, 'recall': 0.46065259117082535, 'f1-score': 0.5245185084004917, 'support': 4168.0}	{'precision': 0.7108843537414966, 'recall': 0.7769516728624535, 'f1-score': 0.7424511545293072, 'support': 2152.0}	{'precision': 0.9222750963063675, 'recall': 0.8822891827444179, 'f1-score': 0.901839131398183, 'support': 9226.0}	{'precision': 0.8349638771824203, 'recall': 0.9189927938374887, 'f1-score': 0.8749654982059067, 'support': 12073.0}	0.8265	{'precision': 0.7692667975544787, 'recall': 0.7597215601537963, 'f1-score': 0.7609435731334722, 'support': 27619.0}	{'precision': 0.820353020671641, 'recall': 0.8264962525797458, 'f1-score': 0.820731174686986, 'support': 27619.0}
No log	3.0	123	0.4293	{'precision': 0.559539052496799, 'recall': 0.6290786948176583, 'f1-score': 0.5922746781115881, 'support': 4168.0}	{'precision': 0.7270703472840605, 'recall': 0.7588289962825279, 'f1-score': 0.7426102773988177, 'support': 2152.0}	{'precision': 0.9509253731343283, 'recall': 0.8632126598742683, 'f1-score': 0.9049485824669052, 'support': 9226.0}	{'precision': 0.8691520467836257, 'recall': 0.8863579889008532, 'f1-score': 0.877670699200328, 'support': 12073.0}	0.8299	{'precision': 0.7766717049247034, 'recall': 0.7843695849688269, 'f1-score': 0.7793760592944097, 'support': 27619.0}	{'precision': 0.8386735331300186, 'recall': 0.8298634997646548, 'f1-score': 0.8331899108807915, 'support': 27619.0}
No log	4.0	164	0.4211	{'precision': 0.6451208032632569, 'recall': 0.4932821497120921, 'f1-score': 0.559075458871516, 'support': 4168.0}	{'precision': 0.78687761749651, 'recall': 0.7857806691449815, 'f1-score': 0.786328760753313, 'support': 2152.0}	{'precision': 0.9139459459459459, 'recall': 0.9163234337741166, 'f1-score': 0.915133145702533, 'support': 9226.0}	{'precision': 0.849535793754316, 'recall': 0.917087716391949, 'f1-score': 0.8820202342069625, 'support': 12073.0}	0.8426	{'precision': 0.7988700401150072, 'recall': 0.7781184922557848, 'f1-score': 0.7856393998835811, 'support': 27619.0}	{'precision': 0.8353211584831782, 'recall': 0.8426445562837177, 'f1-score': 0.836889630165822, 'support': 27619.0}
No log	5.0	205	0.4516	{'precision': 0.5637651821862348, 'recall': 0.6681861804222649, 'f1-score': 0.6115502854633289, 'support': 4168.0}	{'precision': 0.7106579453636014, 'recall': 0.858271375464684, 'f1-score': 0.7775205219953694, 'support': 2152.0}	{'precision': 0.930641011298803, 'recall': 0.901690873618036, 'f1-score': 0.9159372419488027, 'support': 9226.0}	{'precision': 0.9066511085180864, 'recall': 0.8366603164085149, 'f1-score': 0.8702507107779788, 'support': 12073.0}	0.8346	{'precision': 0.7779288118416814, 'recall': 0.8162021864783748, 'f1-score': 0.79381469004637, 'support': 27619.0}	{'precision': 0.8476484297460557, 'recall': 0.8346428183496868, 'f1-score': 0.8392461558560188, 'support': 27619.0}
No log	6.0	246	0.4529	{'precision': 0.5875982161817795, 'recall': 0.6638675623800384, 'f1-score': 0.6234088092824153, 'support': 4168.0}	{'precision': 0.7763975155279503, 'recall': 0.8131970260223048, 'f1-score': 0.7943713118474807, 'support': 2152.0}	{'precision': 0.9178008209116439, 'recall': 0.9209841751571646, 'f1-score': 0.9193897424799826, 'support': 9226.0}	{'precision': 0.9001579224425338, 'recall': 0.8498301996189845, 'f1-score': 0.8742703762089388, 'support': 12073.0}	0.8427	{'precision': 0.7954886187659769, 'recall': 0.8119697407946231, 'f1-score': 0.8028600599547043, 'support': 27619.0}	{'precision': 0.8492397910801023, 'recall': 0.8426807632426953, 'f1-score': 0.8452590968635983, 'support': 27619.0}
No log	7.0	287	0.5035	{'precision': 0.6264514301897479, 'recall': 0.5307101727447217, 'f1-score': 0.574620080529939, 'support': 4168.0}	{'precision': 0.8070770722249152, 'recall': 0.7736988847583643, 'f1-score': 0.7900355871886122, 'support': 2152.0}	{'precision': 0.9218476357267951, 'recall': 0.9128549750704531, 'f1-score': 0.917329266964383, 'support': 9226.0}	{'precision': 0.8542943595313833, 'recall': 0.9120351196885612, 'f1-score': 0.8822209758833427, 'support': 12073.0}	0.8440	{'precision': 0.8024176244182103, 'recall': 0.7823247880655251, 'f1-score': 0.7910514776415692, 'support': 27619.0}	{'precision': 0.8387972595060172, 'recall': 0.8439842137658858, 'f1-score': 0.8403456583559027, 'support': 27619.0}
No log	8.0	328	0.5478	{'precision': 0.5803487276154571, 'recall': 0.5909309021113244, 'f1-score': 0.5855920114122681, 'support': 4168.0}	{'precision': 0.7547730165464573, 'recall': 0.8266728624535316, 'f1-score': 0.7890884896872921, 'support': 2152.0}	{'precision': 0.948723924950472, 'recall': 0.8823975720789075, 'f1-score': 0.9143595215364744, 'support': 9226.0}	{'precision': 0.8638739245798827, 'recall': 0.8899196554294707, 'f1-score': 0.8767033863729091, 'support': 12073.0}	0.8374	{'precision': 0.7869298984230673, 'recall': 0.7974802480183085, 'f1-score': 0.7914358522522359, 'support': 27619.0}	{'precision': 0.8409298617384836, 'recall': 0.8373583402730005, 'f1-score': 0.8385237286921696, 'support': 27619.0}
No log	9.0	369	0.5551	{'precision': 0.5595470519328387, 'recall': 0.6876199616122841, 'f1-score': 0.6170075349838535, 'support': 4168.0}	{'precision': 0.7613151152860803, 'recall': 0.8285315985130112, 'f1-score': 0.7935024477080552, 'support': 2152.0}	{'precision': 0.9396424097483203, 'recall': 0.8943203988727509, 'f1-score': 0.916421391681013, 'support': 9226.0}	{'precision': 0.8944082996307368, 'recall': 0.8426240371075955, 'f1-score': 0.8677442743208086, 'support': 12073.0}	0.8354	{'precision': 0.788728219149494, 'recall': 0.8132739990264104, 'f1-score': 0.7986689121734325, 'support': 27619.0}	{'precision': 0.84861416106056, 'recall': 0.8354031644882146, 'f1-score': 0.8403810802999596, 'support': 27619.0}
No log	10.0	410	0.5735	{'precision': 0.6080889309366131, 'recall': 0.6168426103646834, 'f1-score': 0.6124344926155313, 'support': 4168.0}	{'precision': 0.7508333333333334, 'recall': 0.837360594795539, 'f1-score': 0.7917398945518453, 'support': 2152.0}	{'precision': 0.9385544915640675, 'recall': 0.8923693908519401, 'f1-score': 0.914879431047894, 'support': 9226.0}	{'precision': 0.8762582862754726, 'recall': 0.8868549656257765, 'f1-score': 0.8815247818211758, 'support': 12073.0}	0.8441	{'precision': 0.7934337605273717, 'recall': 0.8083568904094847, 'f1-score': 0.8001446500091116, 'support': 27619.0}	{'precision': 0.8468256644647166, 'recall': 0.8440928346428184, 'f1-score': 0.8450623679377253, 'support': 27619.0}
No log	11.0	451	0.5724	{'precision': 0.5911991199119911, 'recall': 0.6446737044145874, 'f1-score': 0.6167795248479283, 'support': 4168.0}	{'precision': 0.7570093457943925, 'recall': 0.8280669144981413, 'f1-score': 0.7909454061251665, 'support': 2152.0}	{'precision': 0.9376067402937492, 'recall': 0.8925861695209192, 'f1-score': 0.9145427286356822, 'support': 9226.0}	{'precision': 0.8821311887408897, 'recall': 0.8721941522405368, 'f1-score': 0.8771345272803, 'support': 12073.0}	0.8412	{'precision': 0.7919865986852557, 'recall': 0.8093802351685462, 'f1-score': 0.7998505467222692, 'support': 27619.0}	{'precision': 0.8470086415714401, 'recall': 0.8412324848835946, 'f1-score': 0.8436246039246672, 'support': 27619.0}
No log	12.0	492	0.5952	{'precision': 0.6041920545941993, 'recall': 0.5947696737044146, 'f1-score': 0.5994438399226211, 'support': 4168.0}	{'precision': 0.7799544419134397, 'recall': 0.7955390334572491, 'f1-score': 0.7876696572348746, 'support': 2152.0}	{'precision': 0.930126130148454, 'recall': 0.9032083243008888, 'f1-score': 0.9164696178168821, 'support': 9226.0}	{'precision': 0.869600388286685, 'recall': 0.8904166321543941, 'f1-score': 0.8798854102721507, 'support': 12073.0}	0.8427	{'precision': 0.7959682537356945, 'recall': 0.7959834159042366, 'f1-score': 0.7958671313116321, 'support': 27619.0}	{'precision': 0.8427808250509116, 'recall': 0.8426807632426953, 'f1-score': 0.842599380113732, 'support': 27619.0}
0.2759	13.0	533	0.5991	{'precision': 0.6002427184466019, 'recall': 0.5933301343570058, 'f1-score': 0.5967664092664092, 'support': 4168.0}	{'precision': 0.7753721244925575, 'recall': 0.7987918215613383, 'f1-score': 0.7869077592126345, 'support': 2152.0}	{'precision': 0.9312121891104638, 'recall': 0.9009321482766096, 'f1-score': 0.9158219479947113, 'support': 9226.0}	{'precision': 0.8693752023308514, 'recall': 0.8897539965211629, 'f1-score': 0.8794465594170862, 'support': 12073.0}	0.8417	{'precision': 0.7940505585951186, 'recall': 0.7957020251790292, 'f1-score': 0.7947356689727103, 'support': 27619.0}	{'precision': 0.8420921444247412, 'recall': 0.8416669683913248, 'f1-score': 0.8417277778228636, 'support': 27619.0}

Framework versions

• Transformers 4.37.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.0
• Tokenizers 0.15.2