en dataset:cerebras/SlimPajama-627B dataset:EleutherAI/pile license:apache-2.0 region:us
---
datasets:
- cerebras/SlimPajama-627B
- EleutherAI/pile
language:
- en
license: apache-2.0
---

![An eagle flying high up in the sky](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F304f2c7a-fc67-4df4-ba57-c6f38f86826c_2688x1536.png)

### RWKV EagleX 7B v2 Model

> **!Important!: This is not meant to be used with huggingface transformers library**  
> [Use the Hugging Face varient instead, found here (v5-EagleX-v2-7B-HF)](https://huggingface.co/RWKV/v5-EagleX-v2-7B-HF)
> 
> The following is the raw representation of the EagleX 7B v2 model. For use with our own set of trainers
>
> 
> This is not an instruct tune model! (soon...)

## Quickstart with the hugging face transformer library

[See the huggingface version here (v5-EagleX-v2-7B-HF)](huggingface.co/RWKV/v5-EagleX-v2-7B-HF)

```
model = AutoModelForCausalLM.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True).to(torch.float32)
tokenizer = AutoTokenizer.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True)
```

## Evaluation

The following shows the progression of the model from 1.1T trained to 2.25T trained.

|Model                 |Eagle-7B-HF|EagleX-7B-HF-v1|EagleX-7B-HF-v2|
|----------------------|-----------|---------------|---------------|
|Param Count           |7.52 B     |7.52 B         |7.52 B         |
|Tokens Trained        |1.1 T      |1.7 T          |2.25 T         |
|avg_acc               |0.4822     |0.5391         |0.5495         |
|glue (acc)            |0.5752     |0.7463         |0.7439         |
|anli (acc)            |0.3594     |0.4847         |0.5097         |
|mnli (acc)            |0.3802     |0.7928         |0.7884         |
|mnli_mismatch (acc)   |0.3687     |0.7985         |0.784          |
|swag (acc)            |0.568      |0.5814         |0.5905         |
|lambada_standard (acc)|0.685      |0.686          |0.7004         |
|lambada_openai (acc)  |0.7425     |0.7522         |0.7502         |
|mmlu (acc)            |0.3321     |0.4014         |0.438          |
|winogrande (acc)      |0.674      |0.7206         |0.7332         |
|wnli (acc)            |0.4225     |0.4648         |0.493          |
|truthfulqa (acc)      |0.3303     |0.3268         |0.3401         |
|logiqa (acc)          |0.2458     |0.2458         |0.2458         |
|logiqa2 (acc)         |0.2494     |0.2595         |0.2621         |
|sciq (acc)            |0.955      |0.96           |0.93           |
|piqa (acc)            |0.7704     |0.7758         |0.7764         |
|arc_easy (acc)        |0.7382     |0.7555         |0.7445         |
|arc_challenge (acc)   |0.3951     |0.4087         |0.4155         |
|hellaswag (acc)       |0.5264     |0.5411         |0.56           |
|openbookqa (acc)      |0.302      |0.296          |0.304          |
|mathqa (acc)          |0.26       |0.26           |0.2593         |
|arithmetic (acc)      |0.245      |0.0634         |0.1703         |


Compared against other top performing models in the same weight class.

|Model                 |OLMo-7B        |falcon-7b       |Llama-2-7b-hf|EagleX-7B-HF-v2|Mistral-7B-v0.1|
|----------------------|---------------|----------------|-------------|---------------|---------------|
|Param Count           |6.89 B         |6.92 B          |6.74 B       |7.52 B         |7.24 B         |
|Tokens Trained        |2.5 T          |1.5 T           |2 T          |2.25 T         |2 - 7 T?       |
|avg_acc               |0.4578         |0.4775          |0.5045       |0.5495         |0.5676         |
|glue (acc)            |0.474          |0.4578          |0.4289       |0.7439         |0.515          |
|anli (acc)            |0.3478         |0.3541          |0.3697       |0.5097         |0.3803         |
|mnli (acc)            |0.3294         |0.3893          |0.4269       |0.7884         |0.4542         |
|mnli_mismatch (acc)   |0.3348         |0.404           |0.4395       |0.784          |0.4632         |
|swag (acc)            |0.5512         |0.5685          |0.5658       |0.5905         |0.5756         |
|lambada_standard (acc)|0.6396         |0.6868          |0.6808       |0.7004         |0.6944         |
|lambada_openai (acc)  |0.6872         |0.746           |0.7353       |0.7502         |0.7553         |
|mmlu (acc)            |0.2812         |0.2512          |0.4077       |0.438          |0.5964         |
|winogrande (acc)      |0.6725         |0.6709          |0.6914       |0.7332         |0.7364         |
|wnli (acc)            |0.5775         |0.4789          |0.4648       |0.493          |0.5775         |
|truthfulqa (acc)      |0.3015         |0.2826          |0.3205       |0.3401         |0.3537         |
|logiqa (acc)          |0.2335         |0.2151          |0.2535       |0.2458         |0.2427         |
|logiqa2 (acc)         |0.2506         |0.2252          |0.2564       |0.2621         |0.3022         |
|sciq (acc)            |0.927          |0.944           |0.939        |0.93           |0.959          |
|piqa (acc)            |0.7878         |0.7949          |0.7807       |0.7764         |0.8052         |
|arc_easy (acc)        |0.7353         |0.7479          |0.7643       |0.7445         |0.8081         |
|arc_challenge (acc)   |0.3677         |0.4027          |0.4309       |0.4155         |0.5009         |
|hellaswag (acc)       |0.5572         |0.5772          |0.5713       |0.56           |0.6131         |
|openbookqa (acc)      |0.292          |0.306           |0.316        |0.304          |0.33           |
|mathqa (acc)          |0.26           |0.2884          |0.2801       |0.2593         |0.3554         |
|arithmetic (acc)      |0.0069         |0.2367          |0.4703       |0.1703         |0.9004         |


See the following, for the full details on this model: [https://blog.rwkv.com/p/eaglex-v2-soaring-past-llama2-7b](https://blog.rwkv.com/p/eaglex-v2-soaring-past-llama2-7b)


## Links
- [Our wiki](https://wiki.rwkv.com)
- [Full eval data](https://docs.google.com/spreadsheets/d/1CBLU6yKkW-8FMvGD4INO3qjeHZ0qkKnZFcM6n6lWNOs/edit#gid=912381775)
- [Recursal.AI Cloud Platform](https://recursal.ai)
- [HF Gradio Demo](https://huggingface.co/spaces/RWKV/v5-EagleX-v2-7B-gradio)
- [Blog article, detailing our model launch](https://blog.rwkv.com/p/eaglex-v2-soaring-past-llama2-7b)

## Acknowledgement
We are grateful for the help and support from the following key groups:

- [Recursal.ai](https://recursal.ai) team for financing the GPU resources, and managing the training of this foundation model - you can run the Eagle line of RWKV models on their cloud / on-premise platform today.
- EleutherAI for their support, especially in the v5/v6 Eagle/Finch paper
- Linux Foundation AI & Data group for supporting and hosting the RWKV project