Mistral-7B-Instruct-v0.3-JEP

Éste modelo fue afinado con mistralai/Mistral-7B-Instruct-v0.3 sobre el corpus jdavit/colombian-conflict-SQA que tiene información pública de la JEP logrando una función de perdida entre el conjunto de entrenamiento y el de testeo de 0.9339.

Model description

Este es un modelo entrenado sobre el modelo original de mistralai/Mistral-7B-Instruct-v0.3 con el fin de obtner un modelo para un chatbot que responda a preguntas de los casos presentados en la JEP-Colombia. Este es un ejercicio académico realizado por estudiantes de la Univalle.

Intended uses & limitations

More information needed

Training and evaluation data

El datasete jdavit/colombian-conflict-SQA está conformado de 2896 ejemplos de pregunta-respuesta y contexto.

Training procedure

El modelo fue entrenado por 4 horas con: trainable params: 6,815,744 || all params: 7,254,839,296 || trainable%: 0.0939

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.1764	0.1535	100	1.1504
1.0487	0.3070	200	1.0548
0.9853	0.4605	300	1.0175
0.9844	0.6140	400	0.9919
1.011	0.7675	500	0.9780
0.9396	0.9210	600	0.9663
0.9259	1.0737	700	0.9569
0.9444	1.2272	800	0.9483
0.8928	1.3807	900	0.9415
0.9195	1.5342	1000	0.9364
0.8967	1.6876	1100	0.9338
0.927	1.8411	1200	0.9300
0.9417	1.9946	1300	0.9263
0.9198	2.1474	1400	0.9276
0.9108	2.3008	1500	0.9237
0.8971	2.4543	1600	0.9223
0.8758	2.6078	1700	0.9199
0.8681	2.7613	1800	0.9169
0.8557	2.9148	1900	0.9153
0.82	3.0675	2000	0.9161
0.8379	3.2210	2100	0.9170
0.8414	3.3745	2200	0.9161
0.9164	3.5280	2300	0.9141
0.8764	3.6815	2400	0.9101
0.8449	3.8350	2500	0.9094
0.8708	3.9885	2600	0.9088
0.83	4.1412	2700	0.9132
0.7793	4.2947	2800	0.9148
0.8527	4.4482	2900	0.9120
0.7941	4.6017	3000	0.9102
0.8103	4.7552	3100	0.9111
0.7991	4.9087	3200	0.9083
0.7791	5.0614	3300	0.9126
0.8297	5.2149	3400	0.9154
0.739	5.3684	3500	0.9181
0.8456	5.5219	3600	0.9105
0.826	5.6754	3700	0.9135
0.8336	5.8289	3800	0.9127
0.7995	5.9823	3900	0.9134
0.7782	6.1351	4000	0.9207
0.7822	6.2886	4100	0.9170
0.7556	6.4421	4200	0.9182
0.7522	6.5955	4300	0.9213
0.7669	6.7490	4400	0.9168
0.7503	6.9025	4500	0.9173
0.7739	7.0553	4600	0.9217
0.7699	7.2087	4700	0.9293
0.761	7.3622	4800	0.9234
0.7257	7.5157	4900	0.9269
0.7394	7.6692	5000	0.9233
0.7354	7.8227	5100	0.9218
0.8162	7.9762	5200	0.9209
0.7276	8.1289	5300	0.9294
0.7477	8.2824	5400	0.9299
0.7278	8.4359	5500	0.9282
0.6571	8.5894	5600	0.9297
0.7494	8.7429	5700	0.9286
0.767	8.8964	5800	0.9267
0.6792	9.0491	5900	0.9338
0.7053	9.2026	6000	0.9350
0.706	9.3561	6100	0.9351
0.7232	9.5096	6200	0.9334
0.7301	9.6631	6300	0.9332
0.7424	9.8166	6400	0.9344
0.6775	9.9701	6500	0.9339

Framework versions

• PEFT 0.15.2
• Transformers 4.51.3
• Pytorch 2.6.0+cu126
• Datasets 3.5.0
• Tokenizers 0.21.1