Model Name - L3.3-70B-Animus-V2

Character Card & Lore Book

For the best roleplaying experience, it is highly recommended to use the provided character card and lore book. These files help guide the model's persona and provide rich, in-universe context.

Download the Character Card and Lore Book here

Model Description

This is a fine-tuned version of meta-llama/Llama-3.3-70B-Instruct specialized for roleplaying and instruction-following within the Wings of Fire universe. This version represents a significant upgrade in data quality, roleplaying capability, and base model architecture.

The model was first adapted on a 3-million-token dataset extracted from the Wings of Fire book series to build a strong foundation of domain knowledge. It was then fine-tuned for 1 epoch on an expanded and cleaned dataset of conversational and roleplay examples.

The goal of this model is to provide a high-quality, immersive, and lore-accurate conversational experience. It can adopt character personas, answer questions about the world, engage in creative storytelling, portray multiple characters at once, and handle more mature themes from the series.

Training Details

Training Hardware

The model was fine-tuned on a single NVIDIA H100 GPU.

Training Procedure

A QLoRA (Quantized Low-Rank Adaptation) approach was used for efficient fine-tuning, with an optimized process configured using Axolotl.

Training Data

The training process involved two main stages:

1. Domain Adaptation (Pre-training): The base model was adapted to the Wings of Fire universe using the Darkhn/WOF_Pretraining dataset, containing 3 million tokens compiled directly from the book series. This step saturated the model with the specific lore, characters, and writing style of the source material.

2. Instruction & Chat Fine-tuning: The model was fine-tuned for 1 epoch on a mixed dataset of 5,000 examples:

   • Roleplay Scenarios (4,200 examples): From Darkhn/WOF_V3_Combined_Dataset. This new dataset features high-quality, multi-turn roleplay. It was specifically curated to teach the model advanced skills like portraying multiple characters simultaneously and handling the more mature or 'darker' themes (approx. 30% of examples) present in the book series. The data was cleaned to remove formatting artifacts like asterisks.
   • QA & Assistant (800 examples): From Darkhn/WOF_QA_V2. These are instruction-response pairs focused on answering lore questions and following commands within the context of the Wings of Fire world.

Intended Use & Limitations

• Intended Use: This model is intended for creative and roleplaying purposes within the Wings of Fire universe. It is designed for fans of the series and is not a general-purpose chatbot.

• Limitations & Quirks:

  • Performance on tasks outside of its training domain (general knowledge, coding, etc.) is not guaranteed and will likely be poor.
  • The model may "hallucinate" or generate plausible but non-canonical information.
  • Content: The roleplay training data includes more mature and darker themes from the Wings of Fire series, such as character death, conflict, and moral ambiguity. The model is capable of generating content reflecting these themes. It can generate gratuitous or explicit content, as always its up to the user what they do with it.
  • Formatting: The training data was cleaned to remove formatting artifacts like asterisks (*...*) for single word emphasis. The model should now produce cleaner, more narrative-style prose compared to previous versions.
  • Safety: This model has not undergone additional safety alignment beyond what was included in its base Llama 3.3 model. Standard responsible AI practices should be followed.

Recommended Sampler Settings

For optimal performance that balances creativity and coherence, the following default sampler settings are recommended.

• Temperature: 0.8-1.1
• Min_P: 0.02
• DRY Sampler:
  • Multiplier: 0.8
  • Allowed Length: 4
  • Base: 1.75

Acknowledgements

• Credit to Meta for the powerful Llama 3.3 architecture.