Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
datasets:
|
3 |
+
- togethercomputer/RedPajama-Data-V2
|
4 |
+
language:
|
5 |
+
- de
|
6 |
+
pipeline_tag: text-generation
|
7 |
+
library_name: transformers
|
8 |
+
license: other
|
9 |
+
|
10 |
+
---
|
11 |
+
|
12 |
+
# LLäMmlein 1B
|
13 |
+
|
14 |
+
|
15 |
+
LLäMmlein 1B is a German LLaMa model trained from scratch using our adapted [Tinyllama](https://github.com/jzhang38/TinyLlama) codebase on the German portion of [RedPajama V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2).
|
16 |
+
To enhance data quality, we additionally deduplicated the dataset on paragraph level and filtered it using a token-to-word ratio filter. The resulting dataset can be found [here](https://huggingface.co/datasets/LSX-UniWue/LLaMmlein-Dataset).
|
17 |
+
|
18 |
+
We provide three model sizes:
|
19 |
+
|
20 |
+
* [LLäMmlein 7B](https://huggingface.co/LSX-UniWue/LLaMmlein_7B)
|
21 |
+
|
22 |
+
* [LLäMmlein 1B](https://huggingface.co/LSX-UniWue/LLaMmlein_1B) ← You are here
|
23 |
+
|
24 |
+
* [LLäMmlein 120M](https://huggingface.co/LSX-UniWue/LLaMmlein_120M)
|
25 |
+
|
26 |
+
|
27 |
+
Find more details on our page our [page](https://www.informatik.uni-wuerzburg.de/datascience/projects/nlp/llammlein/) and our [preprint](https://arxiv.org/abs/2411.11171)!
|
28 |
+
|
29 |
+
|
30 |
+
### Usage
|
31 |
+
|
32 |
+
You can use LLäMmlein with the `transformers` library.
|
33 |
+
(Optional: install `flash-attn` to achieve highest efficiency.)
|
34 |
+
|
35 |
+
```python
|
36 |
+
from transformers import AutoTokenizer, AutoModelForCausalLM
|
37 |
+
|
38 |
+
model_id = "LSX-UniWue/LLaMmlein_1B"
|
39 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
40 |
+
model = AutoModelForCausalLM.from_pretrained(model_id)
|
41 |
+
```
|
42 |
+
|
43 |
+
|
44 |
+
### Intermediate Checkpoints
|
45 |
+
In addition to the final model checkpoint, we publish intermediate checkpoints throughout the full training process as unique branches in this repository.
|
46 |
+
A specific checkpoint can be loaded like this:
|
47 |
+
|
48 |
+
```python
|
49 |
+
from transformers import AutoTokenizer, AutoModelForCausalLM
|
50 |
+
|
51 |
+
model_id = "LSX-UniWue/LLaMmlein_1B"
|
52 |
+
revision = "iter-00420000-ckpt"
|
53 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
|
54 |
+
model = AutoModelForCausalLM.from_pretrained(model_id, revision=revision)
|
55 |
+
```
|
56 |
+
|
57 |
+
Next to the model itself each branch contains all datapoints that were used to train the model up to that point.
|
58 |
+
In the correspinding folder, named after the checkpoint, you can find several `.log` files (depending on the number of GPUs) of the following format:
|
59 |
+
|
60 |
+
```json
|
61 |
+
{"time": 1739809392.679516,
|
62 |
+
"iter_num": 0,
|
63 |
+
"data_id": ["sha1:EDQMBYDCYBLDAZH3MGYM276BM2DEHPPJ", "sha1:SAJCI75DRHZZFGQORV66NB5FVWUAVLFH", "sha1:7RBZV2MCEM4TUGBBWGTFQAKTWUOGETZU", "sha1:234M32IMLZF7455AKOFWDP6HT6YXAYB4", "sha1:2BIZ7LLSHRK5GUGPZM2GM55APTDKBUG2", "sha1:OF7OI77ZT7ROXGMB6LL4RSRANX7REAYK", "sha1:LGPUOCOV3MKETI5F3IHVGZPD4M26NNJL", "sha1:SHIHUW7FJTP5YHFFV2JZ2CAHUVMKK7XG"],
|
64 |
+
"file_id": [0, 0, 0, 0, 0, 0, 0, 0],
|
65 |
+
"process_rank": 0}
|
66 |
+
```
|
67 |
+
|
68 |
+
|
69 |
+
Note: Our earlier models from the paper, which do not include data logging, are available at:
|
70 |
+
* [LLäMmlein 1B prerelease](https://huggingface.co/LSX-UniWue/LLaMmlein_1B_prerelease)
|
71 |
+
|
72 |
+
* [LLäMmlein 120M prerelease](https://huggingface.co/LSX-UniWue/LLaMmlein_120M_prerelease)
|
73 |
+
|
74 |
+
|
75 |
+
|
76 |
+
### License
|
77 |
+
We release the LLäMmlein models under a research-only RAIL-M license. See [license.md](./license.md) for details.
|