transformers llama text-generation en dataset:sail/regmix-data dataset:sail/regmix-data-sample arxiv:2407.01492 license:mit autotrain_compatible endpoints_compatible region:us
---
datasets:
- sail/regmix-data
- sail/regmix-data-sample
language:
- en
license: mit
---


# Models Trained with Random Mixture

This is a collection of 64 language models, each with approximately 1B parameters, trained on different random mixtures of data. This project aims to validate the generalization capabilities of the RegMix approach (https://huggingface.co/papers/2407.01492) from small-scale (e.g., 1M parameters) to large-scale (e.g., 1B parameters) models.

## Key Features

- **Model Size**: 64 separate models, each with ~1B parameters
- **Training Data**: Random data mixtures on the [RegMix-Data](https://huggingface.co/datasets/sail/regmix-data) dataset
- **Purpose**: To validate the effectiveness of RegMix on identifying high-performing data mixture

## Dataset

The models were trained using the [RegMix-Data](https://huggingface.co/datasets/sail/regmix-data) dataset, which is split into different domains from The Pile dataset.

## Training Hyperparameters

| Hyperparameter | Value |
|:---------------|:------|
| Batch Size | 1M tokens |
| Learning Rate | 4e-4 |
| Minimum Learning Rate | 1e-5 |
| Learning Rate Schedule | Cosine |
| Warmup Ratio | 4% |
| Total Tokens | 25B |

## How to Load a Model

You can load any model using the corresponding branch with the Hugging Face Transformers library:

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("sail/data-mixture-random-1b", revision="model-index-1")
tokenizer = AutoTokenizer.from_pretrained("sail/data-mixture-random-1b", revision="model-index-1")
```

## Data Mixture

The specific data mixture used for training each 1B model can be found in the file `train_config.yaml` in each corresponding model branch.

## Model Variants

To access different model variants, simply change the `revision` parameter in the `from_pretrained` method to the desired model index (e.g., "model-index-2", "model-index-3"), and the maxium index is 64.

## Usage Notes

- These models are primarily intended for research purposes.
- Performance may vary depending on the specific task and domain.

## Citation

If you use these models in your research, please cite the RegMix paper:

```
@article{liu2024regmix,
  title={RegMix: Data Mixture as Regression for Language Model Pre-training},
  author={Liu, Qian and Zheng, Xiaosen and Muennighoff, Niklas and Zeng, Guangtao and Dou, Longxu and Pang, Tianyu and Jiang, Jing and Lin, Min},
  journal={arXiv preprint arXiv:2407.01492},
  year={2024}
}
```

For more information about the RegMix methodology and its applications, please refer to the [original paper](https://huggingface.co/papers/2407.01492).

## Performance

We evaluated each model using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). The performance metric for each task is the average of 0-shot to 5-shot `accnorm` (accuracy normalized, if available) or `acc` (accuracy) scores.

### Table 1: Model Index 1-8

| Task          | Model 1 | Model 2 | Model 3 | Model 4 | Model 5 | Model 6 | Model 7 | Model 8 |
|---------------|---------|---------|---------|---------|---------|---------|---------|---------|
| Social IQA    | 33.27   | 33.33   | 33.62   | 33.53   | 33.49   | 33.56   | 33.62   | 33.55   |
| HellaSwag     | 40.58   | 36.86   | 40.58   | 36.06   | 40.07   | 37.85   | 37.93   | 39.59   |
| PiQA          | 67.29   | 65.14   | 67.97   | 64.66   | 67.03   | 65.36   | 66.00   | 66.55   |
| OpenBookQA    | 28.63   | 27.87   | 29.33   | 29.10   | 29.23   | 28.33   | 29.13   | 28.73   |
| Lambada       | 29.17   | 26.86   | 31.55   | 27.11   | 29.16   | 28.92   | 31.53   | 30.92   |
| SciQ          | 80.68   | 79.98   | 81.05   | 80.80   | 82.40   | 79.88   | 78.67   | 79.70   |
| COPA          | 70.50   | 63.83   | 69.17   | 65.00   | 67.50   | 66.00   | 66.67   | 68.67   |
| RACE          | 29.47   | 30.00   | 32.11   | 28.82   | 31.13   | 30.06   | 29.90   | 30.75   |
| ARC Easy      | 50.03   | 48.72   | 50.01   | 46.64   | 51.06   | 47.46   | 46.75   | 48.39   |
| LogiQA        | 23.76   | 24.17   | 25.29   | 25.29   | 24.55   | 25.96   | 25.45   | 26.32   |
| QQP           | 55.71   | 55.90   | 54.84   | 56.52   | 54.01   | 56.34   | 52.35   | 54.20   |
| WinoGrande    | 51.54   | 51.59   | 51.39   | 50.91   | 53.13   | 52.26   | 51.26   | 51.45   |
| MultiRC       | 52.65   | 53.39   | 51.89   | 50.92   | 49.03   | 53.09   | 53.64   | 50.23   |
| **Average**   | **47.18** | **45.97** | **47.60** | **45.80** | **47.06** | **46.54** | **46.38** | **46.85** |

### Table 2: Model Index 9-16

| Task          | Model 9 | Model 10 | Model 11 | Model 12 | Model 13 | Model 14 | Model 15 | Model 16 |
|---------------|---------|----------|----------|----------|----------|----------|----------|----------|
| Social IQA    | 33.43   | 33.21    | 33.31    | 33.17    | 33.28    | 32.43    | 33.57    | 33.70    |
| HellaSwag     | 40.05   | 35.89    | 39.55    | 39.89    | 38.63    | 36.18    | 39.52    | 35.94    |
| PiQA          | 66.60   | 64.74    | 66.29    | 66.27    | 66.90    | 64.05    | 66.70    | 64.51    |
| OpenBookQA    | 28.87   | 26.60    | 29.33    | 28.73    | 29.40    | 27.87    | 29.67    | 27.83    |
| Lambada       | 31.39   | 27.37    | 30.32    | 30.31    | 31.38    | 26.25    | 29.86    | 26.95    |
| SciQ          | 81.10   | 79.12    | 79.97    | 82.85    | 79.42    | 81.40    | 81.38    | 81.23    |
| COPA          | 67.00   | 64.50    | 66.83    | 69.50    | 67.33    | 65.83    | 69.50    | 66.33    |
| RACE          | 30.57   | 29.63    | 30.49    | 30.85    | 30.35    | 28.66    | 31.21    | 29.57    |
| ARC Easy      | 50.66   | 47.74    | 47.47    | 50.18    | 49.92    | 49.52    | 50.73    | 48.65    |
| LogiQA        | 23.60   | 25.65    | 26.37    | 23.81    | 25.58    | 26.29    | 25.86    | 25.12    |
| QQP           | 54.89   | 54.79    | 54.20    | 55.23    | 53.69    | 57.09    | 53.95    | 54.24    |
| WinoGrande    | 50.83   | 51.84    | 51.05    | 51.83    | 52.12    | 52.00    | 51.01    | 51.82    |
| MultiRC       | 54.18   | 54.48    | 50.17    | 52.12    | 51.42    | 52.69    | 51.87    | 53.48    |
| **Average**   | **47.17** | **45.81** | **46.57** | **47.29** | **46.88** | **46.17** | **47.30** | **46.11** |

### Table 3: Model Index 17-24

| Task          | Model 17 | Model 18 | Model 19 | Model 20 | Model 21 | Model 22 | Model 23 | Model 24 |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|
| Social IQA    | 33.89    | 33.31    | 33.53    | 33.38    | 33.75    | 33.24    | 33.56    | 33.71    |
| HellaSwag     | 38.68    | 39.90    | 34.67    | 37.12    | 37.44    | 36.07    | 42.15    | 34.67    |
| PiQA          | 66.83    | 67.39    | 63.33    | 64.83    | 65.00    | 63.68    | 67.80    | 62.99    |
| OpenBookQA    | 28.13    | 30.67    | 28.03    | 29.40    | 27.67    | 27.77    | 29.37    | 25.83    |
| Lambada       | 28.78    | 28.56    | 24.13    | 29.41    | 27.67    | 28.03    | 33.47    | 24.04    |
| SciQ          | 79.60    | 78.83    | 77.42    | 78.98    | 78.95    | 78.72    | 81.83    | 79.12    |
| COPA          | 65.17    | 68.17    | 65.33    | 67.33    | 67.67    | 62.67    | 69.83    | 65.83    |
| RACE          | 28.74    | 30.03    | 29.76    | 29.49    | 30.77    | 29.76    | 31.21    | 27.91    |
| ARC Easy      | 48.86    | 49.42    | 47.90    | 48.30    | 47.88    | 46.68    | 50.92    | 45.24    |
| LogiQA        | 25.91    | 26.34    | 26.24    | 25.76    | 26.11    | 26.24    | 24.17    | 25.91    |
| QQP           | 53.35    | 53.18    | 50.61    | 51.49    | 54.27    | 54.99    | 52.77    | 55.19    |
| WinoGrande    | 52.54    | 51.17    | 52.01    | 51.09    | 52.13    | 52.03    | 52.50    | 50.28    |
| MultiRC       | 51.49    | 52.45    | 55.40    | 54.87    | 51.73    | 49.49    | 50.61    | 50.29    |
| **Average**   | **46.30** | **46.88** | **45.26** | **46.27** | **46.23** | **45.34** | **47.71** | **44.69** |

### Table 4: Model Index 25-32

| Task          | Model 25 | Model 26 | Model 27 | Model 28 | Model 29 | Model 30 | Model 31 | Model 32 |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|
| Social IQA    | 33.51    | 33.40    | 33.59    | 33.52    | 33.53    | 33.49    | 33.16    | 33.56    |
| HellaSwag     | 36.75    | 36.97    | 40.81    | 38.25    | 40.28    | 35.71    | 37.37    | 37.39    |
| PiQA          | 64.09    | 64.74    | 67.97    | 66.15    | 66.88    | 63.84    | 64.47    | 65.05    |
| OpenBookQA    | 29.47    | 28.70    | 29.57    | 29.77    | 29.50    | 29.13    | 29.47    | 28.00    |
| Lambada       | 26.69    | 33.00    | 31.60    | 33.08    | 31.49    | 27.69    | 26.99    | 29.54    |
| SciQ          | 80.03    | 79.17    | 80.12    | 80.22    | 81.92    | 78.23    | 77.42    | 80.87    |
| COPA          | 67.67    | 65.50    | 69.00    | 65.67    | 68.33    | 63.33    | 64.67    | 67.17    |
| RACE          | 30.05    | 30.19    | 30.96    | 30.37    | 30.08    | 29.62    | 30.13    | 29.92    |
| ARC Easy      | 47.50    | 46.90    | 50.26    | 48.57    | 50.55    | 46.96    | 48.77    | 48.79    |
| LogiQA        | 27.24    | 25.55    | 25.86    | 24.37    | 25.32    | 25.12    | 26.40    | 24.30    |
| QQP           | 49.68    | 55.43    | 50.94    | 50.91    | 51.99    | 53.53    | 49.53    | 51.36    |
| WinoGrande    | 51.68    | 52.12    | 51.93    | 51.50    | 52.32    | 51.67    | 52.13    | 52.63    |
| MultiRC       | 51.24    | 51.91    | 50.33    | 52.42    | 52.52    | 54.04    | 52.05    | 53.04    |
| **Average**   | **45.82** | **46.43** | **47.15** | **46.52** | **47.29** | **45.57** | **45.58** | **46.28** |

### Table 5: Model Index 33-40

| Task          | Model 33 | Model 34 | Model 35 | Model 36 | Model 37 | Model 38 | Model 39 | Model 40 |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|
| Social IQA    | 33.48    | 33.28    | 33.35    | 33.29    | 33.63    | 33.61    | 33.21    | 33.61    |
| HellaSwag     | 38.00    | 40.18    | 43.37    | 37.69    | 32.96    | 32.98    | 37.31    | 37.79    |
| PiQA          | 65.30    | 66.68    | 69.04    | 66.46    | 62.25    | 60.17    | 65.24    | 65.32    |
| OpenBookQA    | 29.43    | 30.37    | 30.43    | 27.63    | 26.43    | 26.83    | 27.97    | 28.70    |
| Lambada       | 26.59    | 31.46    | 31.71    | 30.21    | 18.92    | 20.29    | 28.10    | 28.58    |
| SciQ          | 79.82    | 80.58    | 82.13    | 80.83    | 76.73    | 77.90    | 79.12    | 79.60    |
| COPA          | 64.33    | 69.33    | 67.00    | 67.83    | 61.50    | 62.67    | 64.67    | 66.00    |
| RACE          | 30.03    | 30.16    | 32.47    | 30.49    | 29.27    | 28.12    | 30.11    | 30.21    |
| ARC Easy      | 48.86    | 49.88    | 52.22    | 48.32    | 44.86    | 45.54    | 48.15    | 48.86    |
| LogiQA        | 25.91    | 24.30    | 23.35    | 24.96    | 26.19    | 27.68    | 25.47    | 25.37    |
| QQP           | 56.06    | 56.56    | 52.57    | 56.70    | 52.54    | 48.04    | 49.81    | 57.12    |
| WinoGrande    | 50.92    | 50.97    | 52.39    | 52.70    | 52.30    | 51.68    | 51.42    | 52.80    |
| MultiRC       | 53.09    | 49.97    | 52.18    | 49.05    | 53.78    | 52.27    | 51.45    | 55.68    |
| **Average**   | **46.29** | **47.21** | **47.86** | **46.63** | **43.95** | **43.67** | **45.54** | **46.90** |


### Table 6: Model Index 41-48

| Task          | Model 41 | Model 42 | Model 43 | Model 44 | Model 45 | Model 46 | Model 47 | Model 48 |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|
| Social IQA    | 33.49    | 33.43    | 33.07    | 33.28    | 33.44    | 33.08    | 33.78    | 33.17    |
| HellaSwag     | 34.51    | 37.59    | 42.69    | 37.37    | 38.31    | 38.30    | 39.67    | 41.07    |
| PiQA          | 62.24    | 65.58    | 68.05    | 66.62    | 66.54    | 65.52    | 66.98    | 67.21    |
| OpenBookQA    | 27.10    | 28.77    | 28.90    | 28.07    | 28.07    | 27.60    | 31.17    | 29.73    |
| Lambada       | 22.78    | 26.99    | 31.34    | 29.51    | 27.87    | 29.47    | 30.34    | 32.71    |
| SciQ          | 77.78    | 80.25    | 79.47    | 80.25    | 80.70    | 79.72    | 81.35    | 81.77    |
| COPA          | 64.00    | 66.33    | 67.00    | 67.00    | 67.33    | 68.33    | 67.17    | 67.67    |
| RACE          | 28.33    | 28.82    | 30.78    | 30.80    | 30.08    | 30.24    | 30.24    | 30.67    |
| ARC Easy      | 45.48    | 48.64    | 51.49    | 46.99    | 48.79    | 48.05    | 49.58    | 49.49    |
| LogiQA        | 24.83    | 24.96    | 24.76    | 23.25    | 26.06    | 25.55    | 24.32    | 24.68    |
| QQP           | 50.27    | 54.73    | 53.96    | 57.00    | 53.73    | 51.19    | 57.52    | 56.91    |
| WinoGrande    | 51.79    | 51.63    | 51.32    | 50.76    | 53.18    | 52.45    | 50.72    | 52.24    |
| MultiRC       | 54.03    | 53.96    | 48.91    | 50.74    | 53.01    | 50.89    | 47.63    | 53.84    |
| **Average**   | **44.35** | **46.28** | **47.06** | **46.28** | **46.70** | **46.18** | **46.96** | **47.78** |


## Table 7: Model Index 49-56

| Task          | Model 49 | Model 50 | Model 51 | Model 52 | Model 53 | Model 54 | Model 55 | Model 56 |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|
| Social IQA    | 33.53    | 33.74    | 33.37    | 33.41    | 32.96    | 33.88    | 33.75    | 33.79    |
| HellaSwag     | 39.09    | 35.65    | 38.68    | 36.07    | 37.68    | 38.53    | 35.40    | 40.50    |
| PiQA          | 66.81    | 64.58    | 65.68    | 63.99    | 65.85    | 65.76    | 64.51    | 66.89    |
| OpenBookQA    | 29.13    | 27.57    | 28.27    | 29.10    | 29.43    | 28.73    | 28.30    | 29.87    |
| Lambada       | 30.23    | 26.19    | 30.29    | 30.84    | 29.76    | 29.03    | 28.63    | 30.74    |
| SciQ          | 79.90    | 80.83    | 78.40    | 80.03    | 81.38    | 80.92    | 77.75    | 82.07    |
| COPA          | 68.17    | 61.83    | 67.00    | 66.00    | 66.17    | 63.17    | 66.33    | 64.00    |
| RACE          | 31.42    | 29.35    | 30.41    | 31.08    | 30.77    | 29.73    | 30.80    | 31.42    |
| ARC Easy      | 49.54    | 47.71    | 49.02    | 47.64    | 48.38    | 49.36    | 46.96    | 51.22    |
| LogiQA        | 24.99    | 24.58    | 25.32    | 24.91    | 25.17    | 26.22    | 24.63    | 24.91    |
| QQP           | 54.06    | 56.48    | 50.96    | 56.62    | 56.45    | 53.86    | 53.85    | 53.26    |
| WinoGrande    | 50.51    | 50.26    | 51.83    | 51.33    | 52.18    | 51.89    | 51.59    | 50.50    |
| MultiRC       | 50.25    | 54.37    | 50.94    | 52.38    | 51.21    | 55.34    | 54.52    | 50.50    |
| **Average**   | **46.74** | **45.63** | **46.17** | **46.42** | **46.72** | **46.65** | **45.92** | **46.90** |


## Table 8: Model Index 57-64

| Task          | Model 57 | Model 58 | Model 59 | Model 60 | Model 61 | Model 62 | Model 63 | Model 64 |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|
| Social IQA    | 33.24    | 33.30    | 33.56    | 33.54    | 33.42    | 33.84    | 33.32    | 33.55    |
| HellaSwag     | 41.74    | 39.63    | 35.36    | 38.83    | 38.53    | 36.46    | 38.80    | 36.43    |
| PiQA          | 68.07    | 67.31    | 64.44    | 66.38    | 66.50    | 64.74    | 66.54    | 64.87    |
| OpenBookQA    | 29.20    | 29.50    | 28.10    | 27.97    | 27.83    | 27.37    | 28.83    | 27.87    |
| Lambada       | 31.79    | 31.11    | 27.32    | 30.17    | 28.75    | 26.22    | 30.38    | 26.25    |
| SciQ          | 80.42    | 79.83    | 80.85    | 79.60    | 78.93    | 80.05    | 79.50    | 78.65    |
| COPA          | 66.17    | 69.00    | 64.00    | 64.83    | 67.00    | 64.00    | 66.00    | 66.83    |
| RACE          | 31.39    | 29.82    | 29.67    | 30.08    | 29.98    | 29.46    | 30.37    | 29.19    |
| ARC Easy      | 51.14    | 49.24    | 47.13    | 47.88    | 48.20    | 47.09    | 49.09    | 46.90    |
| LogiQA        | 25.19    | 25.93    | 23.68    | 25.17    | 25.70    | 25.52    | 26.50    | 26.65    |
| QQP           | 55.37    | 54.46    | 52.73    | 53.17    | 59.65    | 58.15    | 57.50    | 55.31    |
| WinoGrande    | 53.21    | 51.46    | 50.83    | 52.16    | 52.37    | 51.41    | 51.63    | 51.85    |
| MultiRC       | 53.58    | 52.31    | 52.22    | 53.03    | 50.41    | 52.17    | 52.27    | 51.50    |
| **Average**   | **47.73** | **47.15** | **45.38** | **46.37** | **46.71** | **45.88** | **46.98** | **45.84** |