|
|
--- |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- text generation |
|
|
- pytorch |
|
|
- causal-lm |
|
|
license: mit |
|
|
datasets: |
|
|
- allenai/c4 |
|
|
- HuggingFaceFW/fineweb-edu |
|
|
- togethercomputer/RedPajama-Data-V2 |
|
|
- Muennighoff/natural-instructions |
|
|
- databricks/databricks-dolly-15k |
|
|
- HuggingFaceTB/smollm-corpus |
|
|
- open-phi/textbooks |
|
|
- roneneldan/TinyStories |
|
|
--- |
|
|
|
|
|
# Mixtress 135M |
|
|
|
|
|
## Model Description |
|
|
|
|
|
Mixtress 135M is a transformer model based upon the [Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mixtral) architecture. It is the culmination of approximately 20 weeks of [Kaggle](https://kaggle.com) free hours, and 67 twelve-hour training runs. |
|
|
|
|
|
## Training data |
|
|
|
|
|
Mixtress was trained on a curated sampling of data from the following datasets: |
|
|
|
|
|
- allenai/c4 |
|
|
- HuggingFaceFW/fineweb-edu |
|
|
- togethercomputer/RedPajama-Data-V2 |
|
|
- Muennighoff/natural-instructions |
|
|
- databricks/databricks-dolly-15k |
|
|
- HuggingFaceTB/smollm-corpus |
|
|
- open-phi/textbooks |
|
|
- roneneldan/TinyStories |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
This model was trained for 2.15 billion tokens over 20,000 optimizer steps. It was trained as a masked autoregressive language model, using cross-entropy loss. |
|
|
|
|
|
The final train loss was 1.941, validation loss was 2.206, and perplexity was 9.136. |
|
|
|
|
|
Mixtress was pre-trained and fine-tuned simultaneously. Full reproduction code may be found [at this URL](https://www.kaggle.com/code/luciferianink/pretraining-a-mixtral), or in the Jupyter notebook [in this repository](./pretraining-a-mixtral.ipynb). |
|
|
|
|
|
## Intended Use and Limitations |
|
|
|
|
|
The model is best at what it was pretrained for, which is generating conversational text and answering questions from a prompt. |
|
|
|
|
|
### How to use |
|
|
|
|
|
You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run: |
|
|
|
|
|
```py |
|
|
>>> from transformers import pipeline |
|
|
>>> generator = pipeline('text-generation', model='UNSAFE/Mixtress-135M') |
|
|
>>> generator("In a shocking finding, ", do_sample=True, temperature=0.7, min_length=50) |
|
|
|
|
|
[{'generated_text': 'In a shocking finding, 20 years ago, U.S. President Donald Trump'}] |
|
|
``` |
|
|
|
|
|
## Eval results |
|
|
|
|
|
All evaluations were done using the [Pythia evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness). |
|
|
|
|
|
### Scores |
|
|
|
|
|
| Model and Size | ARC-easy | ARC-challenge | HellaSwag | PiQA | TinyMMLU | TriviaQA | Winogrande | |
|
|
| ------------------------- | ---------- | ------------- | ---------- | ---------- | ---------- | -------- | ---------- | |
|
|
| EleutherAI/gpt-neo-125m | 22.95% | N/A | 30.26% | N/A | N/A | N/A | N/A | |
|
|
| HuggingFaceTB/SmolLM-135M | 43.99% | N/A | 42.30% | 69.60% | 30.23% | 4.11% | 52.70% | |
|
|
| OpenAI/GPT2-137M | 31.09% | N/A | 29.76% | 62.51% | 26.29% | 0.49% | 49.72% | |
|
|
| **UNSAFE/Mixtress-135M** | **29.21%** | **24.57%** | **26.99%** | **52.67%** | **31.71%** | **N/A** | **50.91%** | |
|
|
|
|
|
## Join Us |
|
|
|
|
|
If you would like to chat with us, please join the [Discord](https://discord.gg/8ZmHP8CqUX) server! |
|
|
|
