metadata
tags:
- generated_from_trainer
datasets:
- BramVanroy/alpaca-dolly-dutch
model-index:
- name: 1e-3lr+512tbs@4n4gpus80
results: []
1e-3lr+512tbs@4n4gpus80
This model is a fine-tuned version of ybelkada/falcon-7b-sharded-bf16 on the BramVanroy/alpaca-dolly-dutch dataset. It achieves the following results on the evaluation set:
- Loss: 1.2612
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- distributed_type: multi-GPU
- num_devices: 16
- total_train_batch_size: 512
- total_eval_batch_size: 512
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 20
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.8677 | 0.16 | 20 | 1.6766 |
| 1.5635 | 0.32 | 40 | 1.5643 |
| 1.6353 | 0.48 | 60 | 1.4980 |
| 1.5166 | 0.65 | 80 | 1.4516 |
| 1.4287 | 0.81 | 100 | 1.4096 |
| 1.5791 | 0.97 | 120 | 1.3802 |
| 1.3911 | 1.13 | 140 | 1.3633 |
| 1.356 | 1.29 | 160 | 1.3419 |
| 1.2524 | 1.45 | 180 | 1.3263 |
| 1.4224 | 1.61 | 200 | 1.3056 |
| 1.2266 | 1.77 | 220 | 1.2897 |
| 1.3242 | 1.94 | 240 | 1.2785 |
| 1.03 | 2.1 | 260 | 1.2957 |
| 1.1643 | 2.26 | 280 | 1.2970 |
| 1.1492 | 2.42 | 300 | 1.2779 |
| 1.0679 | 2.58 | 320 | 1.2770 |
| 1.2695 | 2.74 | 340 | 1.2658 |
| 1.0439 | 2.9 | 360 | 1.2612 |
| 0.9453 | 3.06 | 380 | 1.3157 |
| 0.8494 | 3.23 | 400 | 1.3189 |
| 1.0745 | 3.39 | 420 | 1.3073 |
| 0.8679 | 3.55 | 440 | 1.3019 |
| 1.0569 | 3.71 | 460 | 1.2955 |
| 1.0186 | 3.87 | 480 | 1.2890 |
| 0.8413 | 4.03 | 500 | 1.3445 |
Framework versions
- Transformers 4.30.1
- Pytorch 2.0.1+cu117
- Datasets 2.13.1
- Tokenizers 0.13.3