Hugging Face's logo

nicoboss
/
Mantella-Skyrim-Llama-3-8B-Uncensored-Lora

Text Generation
PEFT
Safetensors
Transformers
llama
axolotl
lora
conversational
text-generation-inference
Model card Files
xet
Community
Mantella-Skyrim-Llama-3-8B-Uncensored-Lora / README.md
nicoboss's picture
nicoboss
Upload folder using huggingface_hub
05a4c0f verified
preview code
|
raw
history blame contribute delete
4.48 kB
metadata
library_name: peft
tags:
  - axolotl
  - base_model:adapter:art-from-the-machine/Mantella-Skyrim-Llama-3-8B
  - lora
  - transformers
datasets:
  - ICEPVP8977/Uncensored_Small_Reasoning
pipeline_tag: text-generation
base_model: art-from-the-machine/Mantella-Skyrim-Llama-3-8B
model-index:
  - name: Mantella-Skyrim-Llama-3-8B-Uncensored-Lora
    results: []

Built with Axolotl

See axolotl config

axolotl version: 0.12.0.dev0

base_model: /pool16_2/Mantella-Skyrim-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name

# for use with fft to only train on language model layers
# unfrozen_parameters:
  # - model.language_model.*
  # - lm_head
  # - embed_tokens
load_in_8bit: false
load_in_4bit: false

# these 3 lines are needed for now to handle vision chat templates w images
#skip_prepare_dataset: true
#remove_unused_columns: false
#sample_packing: false

# gemma3 doesn't seem to play nice with ddp
#ddp_find_unused_parameters: true

chat_template: llama3
datasets:
  - path: /root/Uncensored_Reasoner_Small_Chat.json
    type: chat_template
    field_messages: messages
dataset_prepared_path: last_run_prepared_Mantella-Skyrim-Llama-3-8B-uncensored_final
val_set_size: 0.01
output_dir: ./HDD/Mantella-Skyrim-Llama-3-8B-uncensored_final

adapter: lora
# lora_model_dir:
peft_use_rslora: true

sequence_len: 5400
pad_to_sequence_len: false

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true

gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 8
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 0.00004
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
auto_resume_from_checkpoints: true
logging_steps: 1
flash_attention: true
#eager_attention: true

warmup_steps: 50
evals_per_epoch: 2
eval_max_new_tokens: 128
saves_per_epoch: 2
save_total_limit: 100

debug:
weight_decay: 0.0
deepspeed: deepspeed_configs/zero1.json
special_tokens:
   pad_token: <|end_of_text|>

HDD/Mantella-Skyrim-Llama-3-8B-uncensored_final

This model was trained from scratch on the /root/Uncensored_Reasoner_Small_Chat.json dataset. It achieves the following results on the evaluation set:

  • Loss: 1.1322

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 4e-05
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • total_train_batch_size: 8
  • total_eval_batch_size: 8
  • optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 50
  • training_steps: 4495

Training results

Training Loss Epoch Step Validation Loss
No log 0 0 1.8614
1.0431 0.5 281 1.1419
1.206 1.0 562 1.0708
1.061 1.5 843 1.0362
0.7878 2.0 1124 1.0065
0.8042 2.5 1405 0.9940
0.673 3.0 1686 0.9824
0.6364 3.5 1967 1.0085
0.5152 4.0 2248 0.9915
0.4905 4.5 2529 1.0410
0.5221 5.0 2810 1.0166
0.2905 5.5 3091 1.0783
0.3028 6.0 3372 1.0636
0.2572 6.5 3653 1.1089
0.3408 7.0 3934 1.1059
0.2769 7.5 4215 1.1322

Framework versions

  • PEFT 0.16.0
  • Transformers 4.53.2
  • Pytorch 2.7.1+cu128
  • Datasets 4.0.0
  • Tokenizers 0.21.2