Momorami's picture
Update README.md
dd3e02d verified
metadata
license: apache-2.0
base_model: Jiayi-Pan/Tiny-Vicuna-1B
tags:
  - generated_from_trainer
model-index:
  - name: vicuna_1b_stage1
    results: []
datasets:
  - Aeala/ShareGPT_Vicuna_unfiltered
language:
  - en
metrics:
  - accuracy

Built with Axolotl Visualize in Weights & Biases

vicuna_1b_stage1

This model is a fine-tuned version of Jiayi-Pan/Tiny-Vicuna-1B on the Aeala/ShareGPT_Vicuna_unfiltered dataset. It achieves the following results on the evaluation set:

  • Loss: 2.9673

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0005
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 40
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss
3.3461 0.0170 40 3.5421
3.2004 0.0340 80 3.1581
3.0095 0.0510 120 3.0506
2.714 0.0681 160 3.0168
2.9508 0.0851 200 2.9764
2.9774 0.1021 240 2.9598
2.8688 0.1191 280 2.9551
2.8195 0.1361 320 2.9420
2.8471 0.1531 360 2.9328
2.9252 0.1701 400 2.9673

Framework versions

  • Transformers 4.43.0
  • Pytorch 2.3.1
  • Datasets 2.15.0
  • Tokenizers 0.19.1