Hubert-kakeiken-W-closed

This model is a fine-tuned version of rinna/japanese-hubert-base on the ORIGINAL_KAKEIKEN_W_CLOSED - JA dataset. It achieves the following results on the evaluation set:

• Loss: 0.0273
• Wer: 0.9988
• Cer: 1.0164

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 12500
• num_epochs: 40.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
28.44	1.0	824	10.8607	1.0	1.1284
9.2184	2.0	1648	7.5106	1.0	1.1284
7.0297	3.0	2472	4.1902	1.0	1.1284
3.6874	4.0	3296	3.1196	1.0	1.1284
2.7259	5.0	4120	2.2997	1.0	1.1284
2.1431	6.0	4944	1.0496	0.9997	1.0314
0.8891	7.0	5768	0.6380	0.9999	1.0174
0.4891	8.0	6592	0.2589	0.9991	1.0305
0.3675	9.0	7416	0.6099	0.9994	1.0932
0.2744	10.0	8240	0.1262	0.9990	1.0239
0.2278	11.0	9064	0.1107	0.9990	1.0234
0.2148	12.0	9888	0.0672	0.9990	1.0261
0.1927	13.0	10712	0.0592	0.9990	1.0219
0.1723	14.0	11536	0.0800	0.9988	1.0228
0.1725	15.0	12360	0.0518	0.9990	1.0211
0.1628	16.0	13184	0.1359	0.9988	1.0142
0.1567	17.0	14008	0.0444	0.9990	1.0196
0.1436	18.0	14832	0.0421	0.9988	1.0193
0.1351	19.0	15656	0.0375	0.9988	1.0173
0.1454	20.0	16480	0.0304	0.9988	1.0186
0.1252	21.0	17304	0.0568	0.9988	1.0237
0.1233	22.0	18128	0.0291	0.9988	1.0176
0.1179	23.0	18952	0.0271	0.9988	1.0167
0.1108	24.0	19776	0.0279	0.9988	1.0178
0.1031	25.0	20600	0.0351	0.9990	1.0183
0.1006	26.0	21424	0.0441	0.9990	1.0173
0.0946	27.0	22248	0.0306	0.9988	1.0170
0.09	28.0	23072	0.0302	0.9988	1.0177
0.0813	29.0	23896	0.0349	0.9988	1.0178
0.0806	30.0	24720	0.0369	0.9988	1.0179
0.0763	31.0	25544	0.0434	0.9990	1.0165
0.075	32.0	26368	0.0308	0.9988	1.0160
0.0708	33.0	27192	0.0308	0.9988	1.0167
0.0668	34.0	28016	0.0298	0.9988	1.0166
0.0639	35.0	28840	0.0272	0.9988	1.0164
0.0628	36.0	29664	0.0265	0.9988	1.0161
0.0628	37.0	30488	0.0267	0.9988	1.0163
0.0586	38.0	31312	0.0263	0.9988	1.0162
0.058	39.0	32136	0.0280	0.9988	1.0164
0.0588	39.9520	32920	0.0269	0.9988	1.0164

Framework versions

• Transformers 4.48.0
• Pytorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.21.0