nicoboss/MedraN-E4B-Uncensored-Lora

See axolotl config

axolotl version: 0.12.0.dev0

base_model: ./HDD/MedraN_final/merged
processor_type: AutoProcessor

# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name

plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
cut_cross_entropy: true

# for use with fft to only train on language model layers
# unfrozen_parameters:
  # - model.language_model.*
  # - lm_head
  # - embed_tokens
load_in_8bit: false
load_in_4bit: false

# these 3 lines are needed for now to handle vision chat templates w images
skip_prepare_dataset: true
remove_unused_columns: false
sample_packing: false

# gemma3 doesn't seem to play nice with ddp
ddp_find_unused_parameters: true

chat_template: gemma3n
eot_tokens:
  - <end_of_turn>
datasets:
  - path: /root/Uncensored_Reasoner_Small_Chat.json
    type: chat_template
    field_messages: messages
dataset_prepared_path: last_run_prepared_medran_uncensored_final
val_set_size: 0.01
output_dir: ./HDD/MedraN_uncensored_final

adapter: lora
# lora_model_dir:
peft_use_rslora: true

sequence_len: 5400
pad_to_sequence_len: false

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules: 'model.language_model.layers.[\d]+.(mlp|self_attn).(up|down|gate|q|k|v|o)_proj'

gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 10
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 0.00004
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
auto_resume_from_checkpoints: true
logging_steps: 1
#flash_attention: true
eager_attention: true

warmup_steps: 50
evals_per_epoch: 2
eval_max_new_tokens: 128
saves_per_epoch: 2
save_total_limit: 100

debug:
weight_decay: 0.0
use_wandb: true
wandb_project: "MedraN-Uncensored"
wandb_name: "MedraN-Uncensored-bf16-stage1-final"
deepspeed: deepspeed_configs/zero1.json

HDD/MedraN_uncensored_final

This model was trained from scratch on the /root/Uncensored_Reasoner_Small_Chat.json dataset. It achieves the following results on the evaluation set:

• Loss: 0.4726

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 4e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 5619

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0	0	1.9622
1.1262	0.5	281	1.3529
1.3188	1.0	562	1.2051
1.0273	1.5	843	1.1405
1.0187	2.0	1124	1.0350
0.6996	2.5	1405	0.9807
0.8199	3.0	1686	0.8967
0.6026	3.5	1967	0.8557
0.6366	4.0	2248	0.8061
0.6249	4.5	2529	0.7436
0.3654	5.0	2810	0.6693
0.3942	5.5	3091	0.6110
0.2992	6.0	3372	0.5921
0.5288	6.5	3653	0.5716
0.4762	7.0	3934	0.5238
0.3181	7.5	4215	0.5131
0.3146	8.0	4496	0.4884
0.2855	8.5	4777	0.4777
0.3426	9.0	5058	0.4762
0.2711	9.5	5339	0.4726

Framework versions

• PEFT 0.16.0
• Transformers 4.53.2
• Pytorch 2.7.1+cu128
• Datasets 4.0.0
• Tokenizers 0.21.2