mms-1b-all-bigcgen-baseline-52

This model is a fine-tuned version of facebook/mms-1b-all on the BIGCGEN - BEM dataset. It achieves the following results on the evaluation set:

• Loss: 0.5438
• Wer: 0.4959

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 4
• seed: 52
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
8.9619	0.6116	100	4.8601	1.3206
4.7571	1.2202	200	4.1752	1.0002
4.2428	1.8318	300	3.6663	1.0
2.1962	2.4404	400	0.6894	0.6009
0.8446	3.0489	500	0.6600	0.6023
0.8292	3.6606	600	0.6255	0.5557
0.764	4.2691	700	0.6161	0.5316
0.7429	4.8807	800	0.5947	0.5171
0.7554	5.4893	900	0.6004	0.5290
0.7043	6.0979	1000	0.5858	0.5153
0.7267	6.7095	1100	0.5811	0.5124
0.7213	7.3180	1200	0.6464	0.5852
0.702	7.9297	1300	0.5705	0.5067
0.6642	8.5382	1400	0.5744	0.4978
0.6675	9.1468	1500	0.5622	0.5110
0.6523	9.7584	1600	0.5437	0.4959
0.6301	10.3670	1700	0.5551	0.5082
0.6473	10.9786	1800	0.5383	0.5044
0.6306	11.5872	1900	0.5377	0.4949
0.6327	12.1957	2000	0.5403	0.4904
0.6073	12.8073	2100	0.5338	0.4911
0.5931	13.4159	2200	0.5327	0.4985
0.6207	14.0245	2300	0.5257	0.4751
0.61	14.6361	2400	0.5340	0.4935
0.5912	15.2446	2500	0.5324	0.4914
0.5961	15.8563	2600	0.5313	0.4810
0.6097	16.4648	2700	0.5331	0.4935

Framework versions

• Transformers 4.53.0.dev0
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.21.0