Athena-3-3B Model Card

Athena generated this model card!

Model Overview

Athena-3-3B is a 3.09-billion-parameter causal language model fine-tuned from Qwen2.5-3B-Instruct. This model is designed to excel in various natural language processing tasks, offering enhanced reasoning and instruction-following capabilities.

Model Details

Model Developer: Aayan Mishra
Model Type: Causal Language Model
Architecture: Transformer with Rotary Position Embeddings (RoPE), SwiGLU activation, RMSNorm, Attention QKV bias, and tied word embeddings
Parameters: 3.09 billion total (2.77 billion non-embedding)
Layers: 36
Attention Heads: 16 for query and 2 for key-value (Grouped Query Attention)
Vocabulary Size: Approximately 151,646 tokens
Context Length: Supports up to 32,768 tokens
Languages Supported: Primarily English, with basic support for other languages
License: MIT

Training Details

Athena-3-3B was fine-tuned using the Unsloth framework on a single NVIDIA A100 GPU. The fine-tuning process spanned approximately 90 minutes over 60 epochs, utilizing a curated dataset focused on instruction-following and general NLP tasks. This approach aimed to enhance the model's performance in complex reasoning and academic tasks.

Intended Use

Athena-3-3B is designed for a range of applications, including but not limited to:

General NLP Tasks: Engaging in text completion, summarization, and question-answering tasks.
Academic Assistance: Providing support for tutoring, essay composition, and research inquiries.
Data Analysis: Offering insights and interpretations of data-centric queries.

While Athena-3-3B is a powerful tool for various applications, it is not intended for real-time, safety-critical systems or for processing sensitive personal information.

How to Use

To utilize Athena-3-3B, ensure that you have the latest version of the transformers library installed:

pip install transformers

Here's an example of how to load the Athena-3-3B model and generate a response:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Spestly/Athena-3-3B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Explain the concept of entropy in thermodynamics."
messages = [
    {"role": "system", "content": "You are Maverick, an AI assistant designed to be helpful."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Maverick Search usage 🔍

To use this model with Maverick Search, please refer to this repository

Limitations

Users should be aware of the following limitations:

Biases: Athena-3-3B may exhibit biases present in its training data. Users should critically assess outputs, especially in sensitive contexts.
Knowledge Cutoff: The model's knowledge is current up to August 2024. It may not be aware of events or developments occurring after this date.
Language Support: While primarily trained on English data, performance in other languages may be inconsistent.

Acknowledgements

Athena-3-3B builds upon the work of the Qwen team. Gratitude is also extended to the open-source AI community for their contributions to tools and frameworks that facilitated the development of Athena-3-3B.

License

Athena-3-3B is released under the MIT License, permitting wide usage with proper attribution.

Contact

Email: [email protected]

Spestly
/

Athena-3-3B