AiAF/Pretrained-Adapter-SCP-1B-QLoRA

See axolotl config

axolotl version: 0.8.0.dev0

base_model: meta-llama/Llama-3.2-1B-Instruct
# Automatically upload checkpoint and final model to HF
hub_model_id: AiAF/Pretrained-SCP-1B-QLoRA

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: pretraining.jsonl
    type: completion
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/qlora-out/Pretrained-SCP-1B-QLoRA

adapter: qlora
lora_model_dir:

sequence_len: 2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_fan_in_fan_out:
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj

wandb_project: "LLM-Pretraining"
wandb_entity:
wandb_watch: "all"
wandb_name: "Pretrained-SCP-7B-Instruct"
wandb_log_model: "false"

gradient_accumulation_steps: 3
micro_batch_size: 10
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

warmup_steps: 10
evals_per_epoch: 50
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 10
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: "<|end_of_text|>"

Pretrained-SCP-1B-QLoRA

This model is a fine-tuned version of meta-llama/Llama-3.2-1B-Instruct on the pretraining.jsonl dataset. It achieves the following results on the evaluation set:

• Loss: 2.2062

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 10
• eval_batch_size: 10
• seed: 42
• gradient_accumulation_steps: 3
• total_train_batch_size: 30
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
3.0841	0.0020	1	3.0001
2.8276	0.0215	11	2.8296
2.3977	0.0431	22	2.5255
2.2856	0.0646	33	2.4384
2.3735	0.0862	44	2.4082
2.3645	0.1077	55	2.3861
2.1425	0.1292	66	2.3694
2.1541	0.1508	77	2.3545
2.2848	0.1723	88	2.3410
2.2334	0.1939	99	2.3310
2.1278	0.2154	110	2.3213
2.159	0.2369	121	2.3112
2.1407	0.2585	132	2.3006
1.9851	0.2800	143	2.2915
2.0319	0.3016	154	2.2839
2.2373	0.3231	165	2.2755
2.1488	0.3446	176	2.2684
2.0218	0.3662	187	2.2612
1.9256	0.3877	198	2.2552
2.0179	0.4093	209	2.2486
2.0768	0.4308	220	2.2448
2.1068	0.4523	231	2.2408
2.1343	0.4739	242	2.2356
2.2212	0.4954	253	2.2342
2.0442	0.5170	264	2.2302
2.0805	0.5385	275	2.2256
1.9695	0.5601	286	2.2230
1.8559	0.5816	297	2.2206
2.0997	0.6031	308	2.2185
2.0168	0.6247	319	2.2164
1.9304	0.6462	330	2.2148
1.9313	0.6678	341	2.2132
2.1708	0.6893	352	2.2119
2.0596	0.7108	363	2.2109
2.1944	0.7324	374	2.2099
2.0098	0.7539	385	2.2094
2.0344	0.7755	396	2.2087
2.1658	0.7970	407	2.2080
2.1188	0.8185	418	2.2078
1.879	0.8401	429	2.2072
1.9652	0.8616	440	2.2068
2.0429	0.8832	451	2.2066
2.3038	0.9047	462	2.2064
2.153	0.9262	473	2.2063
2.0543	0.9478	484	2.2062
2.0093	0.9693	495	2.2062
2.2437	0.9909	506	2.2062

Framework versions

• PEFT 0.14.0
• Transformers 4.49.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0