mms-1b-swagen-combined-25hrs-model

This model is a fine-tuned version of facebook/mms-1b-all on the SWAGEN - SWA dataset. It achieves the following results on the evaluation set:

• Loss: 0.3033
• Wer: 0.2141

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
16.7552	0.0478	100	3.6402	1.0002
5.9213	0.0957	200	2.5805	1.0548
4.8114	0.1435	300	2.0880	0.9113
3.7837	0.1914	400	1.4961	0.8379
2.4601	0.2392	500	0.8846	0.5581
1.5872	0.2871	600	0.6476	0.3899
1.3127	0.3349	700	0.5520	0.3545
1.1036	0.3828	800	0.5046	0.3386
1.0068	0.4306	900	0.4653	0.3208
1.0488	0.4785	1000	0.4452	0.3176
0.946	0.5263	1100	0.4134	0.3013
0.8942	0.5742	1200	0.4044	0.2825
0.8966	0.6220	1300	0.3830	0.2782
0.8531	0.6699	1400	0.4013	0.2737
0.8402	0.7177	1500	0.3661	0.2673
0.7815	0.7656	1600	0.3558	0.2495
0.724	0.8134	1700	0.3508	0.2478
0.7646	0.8612	1800	0.3463	0.2507
0.7451	0.9091	1900	0.3468	0.2489
0.7511	0.9569	2000	0.3390	0.2434
0.7062	1.0048	2100	0.3440	0.2403
0.6796	1.0526	2200	0.3252	0.2318
0.6866	1.1005	2300	0.3274	0.2289
0.7269	1.1483	2400	0.3232	0.2367
0.7295	1.1962	2500	0.3226	0.2355
0.6511	1.2440	2600	0.3196	0.2330
0.6907	1.2919	2700	0.3197	0.2303
0.6881	1.3397	2800	0.3185	0.2296
0.6519	1.3876	2900	0.3242	0.2311
0.6504	1.4354	3000	0.3187	0.2337
0.6418	1.4833	3100	0.3122	0.2242
0.642	1.5311	3200	0.3115	0.2232
0.6259	1.5789	3300	0.3006	0.2265
0.6786	1.6268	3400	0.3085	0.2205
0.6457	1.6746	3500	0.3053	0.2261
0.6865	1.7225	3600	0.3028	0.2273
0.6241	1.7703	3700	0.2988	0.2216
0.6192	1.8182	3800	0.3017	0.2241
0.609	1.8660	3900	0.2934	0.2185
0.6394	1.9139	4000	0.3008	0.2196
0.571	1.9617	4100	0.2939	0.2172
0.5886	2.0096	4200	0.3011	0.2127
0.5963	2.0574	4300	0.3033	0.2143

Framework versions

• Transformers 4.47.1
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0