gecfdo/Not-WizardLM-2-The-Omega-Directive-7b-Unslop-v2.1-LoRA

See axolotl config

axolotl version: 0.12.0.dev0

base_model: /workspace/data/models/Not-WizardLM-2-7B

#load_in_4bit: true
#load_in_8bit: true
#adapter: qlora
adapter: lora

chat_template_strategy: tokenizer

model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

datasets:
  - path: data/tutu2f_1.03.jsonl
    type: chat_template
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
    roles:
      user: ["user"]
      assistant: ["assistant"]
      system: ["system"]


dataset_prepared_path: last_run_prepared
output_dir: outputs/Mistral-7b-Unslop-v2.1

sequence_len: 5120
pad_to_sequence_len: false

lora_r: 32
lora_alpha: 32
lora_dropout: 0.1
lora_target_linear: true


gradient_accumulation_steps: 1
micro_batch_size: 16

num_epochs: 1
lr_scheduler: cosine
learning_rate: 2e-6

bf16: auto
tf32: false

gradient_checkpointing: true
logging_steps: 5
flash_attention: false

warmup_ratio: 0.05
weight_decay: 0.0

save_steps: 100
save_safetensors: true

do_causal_lm_eval: true
eval_steps: 100
eval_causal_lm_metrics: ["sacrebleu", "perplexity"]

max_grad_norm: 1.0

outputs/Mistral-7b-Unslop-v2.1

This model was trained from scratch on the data/tutu2f_1.03.jsonl dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 63
• training_steps: 1267

Training results

Framework versions

• PEFT 0.16.0
• Transformers 4.54.0
• Pytorch 2.7.1+cu128
• Datasets 4.0.0
• Tokenizers 0.21.2