Text Generation
Transformers
PyTorch
Safetensors
German
llama
text-generation-inference
Julia287 commited on
Commit
81ae414
·
verified ·
1 Parent(s): 198dd22

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +77 -0
README.md ADDED
@@ -0,0 +1,77 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ datasets:
3
+ - togethercomputer/RedPajama-Data-V2
4
+ language:
5
+ - de
6
+ pipeline_tag: text-generation
7
+ library_name: transformers
8
+ license: other
9
+
10
+ ---
11
+
12
+ # LLäMmlein 1B
13
+
14
+
15
+ LLäMmlein 1B is a German LLaMa model trained from scratch using our adapted [Tinyllama](https://github.com/jzhang38/TinyLlama) codebase on the German portion of [RedPajama V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2).
16
+ To enhance data quality, we additionally deduplicated the dataset on paragraph level and filtered it using a token-to-word ratio filter. The resulting dataset can be found [here](https://huggingface.co/datasets/LSX-UniWue/LLaMmlein-Dataset).
17
+
18
+ We provide three model sizes:
19
+
20
+ * [LLäMmlein 7B](https://huggingface.co/LSX-UniWue/LLaMmlein_7B)
21
+
22
+ * [LLäMmlein 1B](https://huggingface.co/LSX-UniWue/LLaMmlein_1B) ← You are here
23
+
24
+ * [LLäMmlein 120M](https://huggingface.co/LSX-UniWue/LLaMmlein_120M)
25
+
26
+
27
+ Find more details on our page our [page](https://www.informatik.uni-wuerzburg.de/datascience/projects/nlp/llammlein/) and our [preprint](https://arxiv.org/abs/2411.11171)!
28
+
29
+
30
+ ### Usage
31
+
32
+ You can use LLäMmlein with the `transformers` library.
33
+ (Optional: install `flash-attn` to achieve highest efficiency.)
34
+
35
+ ```python
36
+ from transformers import AutoTokenizer, AutoModelForCausalLM
37
+
38
+ model_id = "LSX-UniWue/LLaMmlein_1B"
39
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
40
+ model = AutoModelForCausalLM.from_pretrained(model_id)
41
+ ```
42
+
43
+
44
+ ### Intermediate Checkpoints
45
+ In addition to the final model checkpoint, we publish intermediate checkpoints throughout the full training process as unique branches in this repository.
46
+ A specific checkpoint can be loaded like this:
47
+
48
+ ```python
49
+ from transformers import AutoTokenizer, AutoModelForCausalLM
50
+
51
+ model_id = "LSX-UniWue/LLaMmlein_1B"
52
+ revision = "iter-00420000-ckpt"
53
+ tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)
54
+ model = AutoModelForCausalLM.from_pretrained(model_id, revision=revision)
55
+ ```
56
+
57
+ Next to the model itself each branch contains all datapoints that were used to train the model up to that point.
58
+ In the correspinding folder, named after the checkpoint, you can find several `.log` files (depending on the number of GPUs) of the following format:
59
+
60
+ ```json
61
+ {"time": 1739809392.679516,
62
+ "iter_num": 0,
63
+ "data_id": ["sha1:EDQMBYDCYBLDAZH3MGYM276BM2DEHPPJ", "sha1:SAJCI75DRHZZFGQORV66NB5FVWUAVLFH", "sha1:7RBZV2MCEM4TUGBBWGTFQAKTWUOGETZU", "sha1:234M32IMLZF7455AKOFWDP6HT6YXAYB4", "sha1:2BIZ7LLSHRK5GUGPZM2GM55APTDKBUG2", "sha1:OF7OI77ZT7ROXGMB6LL4RSRANX7REAYK", "sha1:LGPUOCOV3MKETI5F3IHVGZPD4M26NNJL", "sha1:SHIHUW7FJTP5YHFFV2JZ2CAHUVMKK7XG"],
64
+ "file_id": [0, 0, 0, 0, 0, 0, 0, 0],
65
+ "process_rank": 0}
66
+ ```
67
+
68
+
69
+ Note: Our earlier models from the paper, which do not include data logging, are available at:
70
+ * [LLäMmlein 1B prerelease](https://huggingface.co/LSX-UniWue/LLaMmlein_1B_prerelease)
71
+
72
+ * [LLäMmlein 120M prerelease](https://huggingface.co/LSX-UniWue/LLaMmlein_120M_prerelease)
73
+
74
+
75
+
76
+ ### License
77
+ We release the LLäMmlein models under a research-only RAIL-M license. See [license.md](./license.md) for details.