File size: 3,098 Bytes
7368704 d605957 7368704 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
---
language:
- en
tags:
- text generation
- pytorch
- causal-lm
license: mit
datasets:
- allenai/c4
- HuggingFaceFW/fineweb-edu
- togethercomputer/RedPajama-Data-V2
- Muennighoff/natural-instructions
- databricks/databricks-dolly-15k
- HuggingFaceTB/smollm-corpus
- open-phi/textbooks
- roneneldan/TinyStories
---
# Mixtress 135M
## Model Description
Mixtress 135M is a transformer model based upon the [Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mixtral) architecture. It is the culmination of approximately 20 weeks of [Kaggle](https://kaggle.com) free hours, and 67 twelve-hour training runs.
## Training data
Mixtress was trained on a curated sampling of data from the following datasets:
- allenai/c4
- HuggingFaceFW/fineweb-edu
- togethercomputer/RedPajama-Data-V2
- Muennighoff/natural-instructions
- databricks/databricks-dolly-15k
- HuggingFaceTB/smollm-corpus
- open-phi/textbooks
- roneneldan/TinyStories
## Training procedure
This model was trained for 2.15 billion tokens over 20,000 optimizer steps. It was trained as a masked autoregressive language model, using cross-entropy loss.
The final train loss was 1.941, validation loss was 2.206, and perplexity was 9.136.
Mixtress was pre-trained and fine-tuned simultaneously. Full reproduction code may be found [at this URL](https://www.kaggle.com/code/luciferianink/pretraining-a-mixtral), or in the Jupyter notebook [in this repository](./pretraining-a-mixtral.ipynb).
## Intended Use and Limitations
The model is best at what it was pretrained for, which is generating conversational text and answering questions from a prompt.
### How to use
You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:
```py
>>> from transformers import pipeline
>>> generator = pipeline('text-generation', model='UNSAFE/Mixtress-135M')
>>> generator("In a shocking finding, ", do_sample=True, temperature=0.7, min_length=50)
[{'generated_text': 'In a shocking finding, 20 years ago, U.S. President Donald Trump'}]
```
## Eval results
All evaluations were done using the [Pythia evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness).
### Scores
| Model and Size | ARC-easy | ARC-challenge | HellaSwag | PiQA | TinyMMLU | TriviaQA | Winogrande |
| ------------------------- | ---------- | ------------- | ---------- | ---------- | ---------- | -------- | ---------- |
| EleutherAI/gpt-neo-125m | 22.95% | N/A | 30.26% | N/A | N/A | N/A | N/A |
| HuggingFaceTB/SmolLM-135M | 43.99% | N/A | 42.30% | 69.60% | 30.23% | 4.11% | 52.70% |
| OpenAI/GPT2-137M | 31.09% | N/A | 29.76% | 62.51% | 26.29% | 0.49% | 49.72% |
| **UNSAFE/Mixtress-135M** | **29.21%** | **24.57%** | **26.99%** | **52.67%** | **31.71%** | **N/A** | **50.91%** |
## Join Us
If you would like to chat with us, please join the [Discord](https://discord.gg/8ZmHP8CqUX) server!
|