transformers safetensors llama text-generation llama-3.1 instruction-tuned conversational en dataset:OpenAssistant/oasst1 dataset:databricks/databricks-dolly-15k dataset:Open-Orca/OpenOrca dataset:mlabonne/open-perfectblend dataset:tatsu-lab/alpaca base_model:meta-llama/Llama-3.1-8B-Instruct base_model:finetune:meta-llama/Llama-3.1-8B-Instruct license:llama3.1 model-index autotrain_compatible text-generation-inference endpoints_compatible region:us
---
base_model: meta-llama/Llama-3.1-8B-Instruct
datasets:
- OpenAssistant/oasst1
- databricks/databricks-dolly-15k
- Open-Orca/OpenOrca
- mlabonne/open-perfectblend
- tatsu-lab/alpaca
language: en
library_name: transformers
license: llama3.1
pipeline_tag: text-generation
tags:
- llama-3.1
- instruction-tuned
model-index:
- name: utkmst/chimera-beta-test2-lora-merged
  results:
  - task:
      type: text-generation
    dataset:
      name: Overall Leaderboard
      type: leaderboard
    metrics:
    - type: acc_norm
      value: 0.444
      name: acc_norm
      verified: true
    - type: acc
      value: 0.2992
      name: acc
      verified: true
    - type: exact_match
      value: 0.0951
      name: exact_match
      verified: true
  - task:
      type: text-generation
    dataset:
      name: BBH (Big Bench Hard)
      type: bbh
    metrics:
    - type: acc_norm
      value: 0.4773
      name: acc_norm
      verified: true
  - task:
      type: text-generation
    dataset:
      name: GPQA (Google-Patched Question Answering)
      type: gpqa
    metrics:
    - type: acc_norm
      value: 0.3036
      name: acc_norm
      verified: true
  - task:
      type: text-generation
    dataset:
      name: Math
      type: math
    metrics:
    - type: exact_match
      value: 0.0951
      name: exact_match
      verified: true
  - task:
      type: text-generation
    dataset:
      name: MMLU-Pro
      type: mmlu_pro
    metrics:
    - type: acc
      value: 0.2992
      name: acc
      verified: true
  - task:
      type: text-generation
    dataset:
      name: MUSR (Multi-Step Reasoning)
      type: musr
    metrics:
    - type: acc_norm
      value: 0.4113
      name: acc_norm
      verified: true
---

# utkmst/chimera-beta-test2-lora-merged

## Model Description
This model is a fine-tuned version of Meta's Llama-3.1-8B-Instruct model, created through LoRA fine-tuning on multiple instruction datasets, followed by merging the adapter weights with the base model.

## Architecture
- **Base Model**: meta-llama/Llama-3.1-8B-Instruct
- **Size**: 8.03B parameters
- **Type**: Decoder-only transformer
- **Format**: SafeTensors (full precision)

## Training Details
- **Training Method**: LoRA fine-tuning followed by adapter merging
- **LoRA Configuration**:
  - Rank: 8 
  - Alpha: 16 
  - Trainable modules: Attention layers and feed-forward networks
- **Training Hyperparameters**:
  - Learning rate: 2e-4 
  - Batch size: 2 
  - Training epochs: 1 
  - Optimizer: AdamW with constant scheduler

## Intended Use
This model is designed for:
- General purpose assistant capabilities
- Question answering and knowledge retrieval
- Creative content generation
- Instructional guidance

## Limitations
- Base model limitations including potential hallucinations and factual inaccuracies
- Limited context window compared to larger models
- Knowledge cutoff from the base Llama-3.1 model
- May exhibit biases present in training data
- Performance on specialized tasks may vary

## Usage with Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("utkmst/chimera-beta-test2-lora-merged")
tokenizer = AutoTokenizer.from_pretrained("utkmst/chimera-beta-test2-lora-merged")
```
## License

This model inherits the license from Meta's Llama 3.1.