gecfdo/MS3.2-The-Omega-Directive-24B-Unslop-v2.0-EXL2-LoRA

See axolotl config

axolotl version: 0.11.0.dev0

base_model: /workspace/data/models/mistral-small-3.2-24B-instruct-2506-text-only

load_in_4bit: true

#chat_template: mistral_v7_tekken
chat_template_strategy: tokenizer

model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
#tokenizer_use_mistral_common: true
trust_remote_code: true

datasets:
  - path: data/tutu2f.jsonl
    type: chat_template
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
    roles:
      user: ["user"]
      assistant: ["assistant"]
      system: ["system"]


dataset_prepared_path: last_run_prepared
output_dir: outputs/Mistral-Small-3.2-24B-Unslop-v2.0


adapter: qlora

sequence_len: 5120
pad_to_sequence_len: false

lora_r: 32
lora_alpha: 32
lora_dropout: 0.1
lora_target_linear: true


gradient_accumulation_steps: 4
micro_batch_size: 4
num_epochs: 1
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 2e-6

bf16: auto
tf32: false

gradient_checkpointing: true
logging_steps: 5
flash_attention: false

warmup_ratio: 0.05
weight_decay: 0.0

save_steps: 100
save_safetensors: true

do_causal_lm_eval: true
eval_steps: 100
eval_causal_lm_metrics: ["sacrebleu", "perplexity"]

max_grad_norm: 1.0

outputs/Mistral-Small-3.2-24B-Unslop-v2.0

This model was trained from scratch on the data/tutu2f.jsonl dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 68
• training_steps: 1372

Training results

Framework versions

• PEFT 0.15.2
• Transformers 4.52.4
• Pytorch 2.7.1+cu128
• Datasets 3.6.0
• Tokenizers 0.21.1