AndyGulp/dec-4b-sft-full

See axolotl config

axolotl version: 0.13.0.dev0

base_model: Qwen/Qwen3-4B-Instruct-2507

plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true

strict: false

datasets:
  - path: alpaca_df.jsonl
    type: alpaca
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/

pad_to_sequence_len: true
sequence_len: 8192
sample_packing: true

wandb_project: qwen3-4b-sft
wandb_name: qwen3-4b-sft

gradient_accumulation_steps: 2
micro_batch_size: 8
num_epochs: 1
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 2e-6

bf16: true
torch_compile: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
resume_from_checkpoint:
logging_steps: 5
flash_attention: true
load_best_model_at_end: true
evals_per_epoch: 20
saves_per_epoch: 5
warmup_ratio: 0.1
weight_decay: 0.0
fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Qwen3DecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true
special_tokens:

outputs/

This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 on the alpaca_df.jsonl dataset. It achieves the following results on the evaluation set:

• Loss: 0.8415
• Memory/max Active (gib): 8.89
• Memory/max Allocated (gib): 8.89
• Memory/device Reserved (gib): 26.02

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-06
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 9
• training_steps: 95

Training results

Training Loss	Epoch	Step	Validation Loss	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	2.0265	7.27	7.27	8.14
2.0249	0.0524	5	1.9972	8.89	8.89	26.06
1.9435	0.1047	10	1.7351	8.89	8.89	26.02
1.5837	0.1571	15	1.3767	8.89	8.89	26.02
1.2933	0.2094	20	1.1584	8.89	8.89	26.02
1.1102	0.2618	25	1.0479	8.89	8.89	26.02
1.0265	0.3141	30	0.9719	8.89	8.89	26.02
0.9662	0.3665	35	0.9244	8.89	8.89	26.02
0.9272	0.4188	40	0.8964	8.89	8.89	26.02
0.8899	0.4712	45	0.8775	8.89	8.89	26.02
0.8913	0.5236	50	0.8638	8.89	8.89	26.02
0.886	0.5759	55	0.8538	8.89	8.89	26.02
0.8618	0.6283	60	0.8481	8.89	8.89	26.02
0.8713	0.6806	65	0.8450	8.89	8.89	26.02
0.8534	0.7330	70	0.8426	8.89	8.89	26.02
0.8685	0.7853	75	0.8417	8.89	8.89	26.02
0.86	0.8377	80	0.8413	8.89	8.89	26.02
0.8583	0.8901	85	0.8412	8.89	8.89	26.02
0.853	0.9424	90	0.8411	8.89	8.89	26.02
0.8562	0.9948	95	0.8415	8.89	8.89	26.02

Framework versions

• Transformers 4.57.1
• Pytorch 2.8.0+cu128
• Datasets 4.3.0
• Tokenizers 0.22.1