miniLM_finetuned_Emotion_2024_06_17

This model is a fine-tuned version of microsoft/MiniLM-L12-H384-uncased on the emotion dataset. It achieves the following results on the evaluation set:

• Loss: 0.4059
• F1: 0.9350

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	F1
1.3684	1.0	250	1.0416	0.5803
0.8635	2.0	500	0.6225	0.8729
0.5165	3.0	750	0.3755	0.9130
0.3319	4.0	1000	0.2792	0.9256
0.2494	5.0	1250	0.2474	0.9252
0.1914	6.0	1500	0.2182	0.9290
0.156	7.0	1750	0.2140	0.9307
0.1435	8.0	2000	0.1807	0.9351
0.1258	9.0	2250	0.1830	0.9353
0.1128	10.0	2500	0.1655	0.9404
0.1023	11.0	2750	0.1968	0.9339
0.0967	12.0	3000	0.1816	0.9333
0.0914	13.0	3250	0.1840	0.9338
0.0818	14.0	3500	0.2094	0.9316
0.0755	15.0	3750	0.1945	0.9345
0.0718	16.0	4000	0.2040	0.9325
0.0641	17.0	4250	0.2230	0.9369
0.0613	18.0	4500	0.2349	0.9332
0.0556	19.0	4750	0.2530	0.9249
0.0521	20.0	5000	0.2334	0.9376
0.0526	21.0	5250	0.2531	0.9306
0.0423	22.0	5500	0.2336	0.9383
0.039	23.0	5750	0.2848	0.9352
0.0435	24.0	6000	0.2955	0.9363
0.0371	25.0	6250	0.3075	0.9362
0.0338	26.0	6500	0.2910	0.9339
0.0319	27.0	6750	0.3133	0.9343
0.0305	28.0	7000	0.3106	0.9344
0.0254	29.0	7250	0.3155	0.9370
0.0288	30.0	7500	0.3310	0.9339
0.0228	31.0	7750	0.3463	0.9364
0.0224	32.0	8000	0.3618	0.9353
0.0207	33.0	8250	0.3720	0.9347
0.022	34.0	8500	0.3672	0.9374
0.0222	35.0	8750	0.3525	0.9388
0.0197	36.0	9000	0.3848	0.9384
0.0196	37.0	9250	0.3722	0.9369
0.0175	38.0	9500	0.3490	0.9350
0.0168	39.0	9750	0.3539	0.9365
0.0167	40.0	10000	0.3590	0.9391
0.0144	41.0	10250	0.3824	0.9382
0.0164	42.0	10500	0.3973	0.9322
0.0124	43.0	10750	0.3892	0.9372
0.012	44.0	11000	0.4102	0.9333
0.0142	45.0	11250	0.3921	0.9366
0.012	46.0	11500	0.3925	0.9361
0.0097	47.0	11750	0.3924	0.9360
0.0107	48.0	12000	0.3952	0.9330
0.0093	49.0	12250	0.4067	0.9360
0.0104	50.0	12500	0.4059	0.9350

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1