Dans-DiscountModels/Dans-Instruct-Mix-8b-ChatML-V0.1.1

See axolotl config

axolotl version: 0.4.1

base_model: Dans-DiscountModels/Meta-Llama-3.1-8B-ChatML
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

trust_remote_code:

# wandb configuration
wandb_project: l3.1-8b-dans-instruct
wandb_watch:
wandb_run_id:
wandb_log_model:

# push checkpoints to hub
hub_model_id: Dans-DiscountModels/Dans-L3.1-Test
# how to push checkpoints to hub
# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy
hub_strategy: every_save
# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets
# Required to be true when used in combination with `push_dataset_to_hub`
hf_use_auth_token: true

# where to save the finished model to
output_dir: ./l3.1-8b-dans-instruct

# dataset settings (local or huggingface repo)
datasets:
  - path: PocketDoc/Dans-MemoryCore-CoreCurriculum-Small
    type: dan-chat
  - path: AquaV/Energetic-Materials-Sharegpt
    type: dan-chat
  - path: AquaV/Chemical-Biological-Safety-Applications-Sharegpt
    type: dan-chat
  - path: AquaV/US-Army-Survival-Sharegpt
    type: dan-chat
  - path: AquaV/Resistance-Sharegpt
    type: dan-chat
  - path: AquaV/Interrogation-Sharegpt
    type: dan-chat
  - path: AquaV/Multi-Environment-Operations-Sharegpt
    type: dan-chat
  - path: PocketDoc/Dans-Mathmaxx
    type: dan-chat
  - path: PocketDoc/Dans-Benchmaxx
    type: dan-chat
  - path: PocketDoc/Dans-Codemaxx
    type: dan-chat
  - path: PocketDoc/Dans-Taskmaxx
    type: dan-chat
  - path: PocketDoc/Dans-Toolmaxx
    type: dan-chat
  - path: PocketDoc/Dans-ASCIIMaxx-Wordart
    type: dan-chat
  - path: PocketDoc/Dans-Prosemaxx-Gutenberg
    type: dan-chat
  - path: PocketDoc/Dans-Prosemaxx-Cowriter-XS
    type: dan-chat
  - path: PocketDoc/Dans-Prosemaxx-Adventure
    type: dan-chat
  - path: PocketDoc/Dans-Prosemaxx-Opus-Writing
    type: dan-chat
  - path: PocketDoc/Dans-Assistantmaxx-Sharegpt
    type: dan-chat
  - path: PocketDoc/Dans-Assistantmaxx-OpenAssistant2
    type: dan-chat
  - path: PocketDoc/Dans-Assistantmaxx-Opus-instruct-1
    type: dan-chat
  - path: PocketDoc/Dans-Assistantmaxx-Opus-instruct-2
    type: dan-chat
  - path: PocketDoc/Dans-Assistantmaxx-Opus-instruct-3
    type: dan-chat
  - path: PocketDoc/DansTestYard
    type: completion

chat_template: chatml

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

load_in_8bit: false
load_in_4bit: false
strict: false

dataset_prepared_path: ./l3.1-8b-dans-instruct-data
val_set_size: 0.01

sequence_len: 8192

sample_packing: true
eval_sample_packing: true

pad_to_sequence_len: true

gradient_checkpointing: true

gradient_accumulation_steps: 32
micro_batch_size: 1

num_epochs: 3

optimizer: adamw_torch

lr_scheduler: cosine
learning_rate: 0.0000015
cosine_min_lr_ratio: 

adam_beta1: 0.9
adam_beta2: 0.95
adam_epsilon: 0.00000001
weight_decay: 0.05

train_on_inputs: false
group_by_length: true

bf16: true
fp16: false
tf32: false

early_stopping_patience:

resume_from_checkpoint: 
auto_resume_from_checkpoints: 

local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_ratio: 0.1
evals_per_epoch: 10
eval_table_size:
eval_max_new_tokens:
saves_per_epoch: 2

debug: false

deepspeed:
fsdp:
fsdp_config:

special_tokens:
  pad_token: <|finetune_right_pad_id|>
  eos_token: <|im_end|>

Dans-L3.1-Test

This model is a fine-tuned version of Dans-DiscountModels/Meta-Llama-3.1-8B-ChatML on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.1721

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1.5e-06
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 67
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.2037	0.0045	1	1.3169
1.4215	0.1026	23	1.3118
1.2555	0.2053	46	1.2826
1.2191	0.3079	69	1.2334
1.1307	0.4106	92	1.2111
1.1966	0.5132	115	1.1994
1.1934	0.6159	138	1.1920
1.2416	0.7185	161	1.1875
1.1793	0.8212	184	1.1845
1.2027	0.9238	207	1.1820
1.1983	1.0244	230	1.1798
1.2347	1.1269	253	1.1783
1.0298	1.2294	276	1.1767
1.1187	1.3318	299	1.1757
1.2363	1.4343	322	1.1748
1.1663	1.5368	345	1.1742
0.939	1.6393	368	1.1735
1.1537	1.7418	391	1.1732
0.9808	1.8443	414	1.1728
1.1022	1.9468	437	1.1726
0.9958	2.0472	460	1.1725
1.2978	2.1497	483	1.1723
1.4147	2.2523	506	1.1721
1.2451	2.3548	529	1.1721
1.1376	2.4573	552	1.1721
1.1449	2.5598	575	1.1721
1.2037	2.6623	598	1.1720
1.2065	2.7649	621	1.1722
1.0512	2.8674	644	1.1721
1.1381	2.9699	667	1.1721

Framework versions

• Transformers 4.45.0.dev0
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1