Improve model card with library name and primary paper link (#3)
Browse files- Improve model card with library name and primary paper link (2429b4407e5c2a8d9666c3f6d604ec7b61483206)
Co-authored-by: Niels Rogge <[email protected]>
README.md
CHANGED
@@ -1,21 +1,23 @@
|
|
1 |
---
|
2 |
-
language:
|
3 |
-
- fi
|
4 |
-
license: apache-2.0
|
5 |
-
tags:
|
6 |
-
- finnish
|
7 |
-
- llama
|
8 |
datasets:
|
9 |
- Finnish-NLP/mc4_3.1.0_fi_cleaned
|
10 |
- Finnish-NLP/oscar_2301_fi_cleaned
|
11 |
- Finnish-NLP/Reddit_fi_2006_2022
|
12 |
- Finnish-NLP/wikipedia_20230501_fi_cleaned
|
13 |
- intfloat/multilingual_cc_news
|
14 |
-
|
|
|
|
|
15 |
pipeline_tag: text-generation
|
16 |
-
|
|
|
|
|
|
|
|
|
17 |
---
|
18 |
|
|
|
|
|
19 |
# Llama-3b for Finnish
|
20 |
|
21 |
Pretrained Llama model on Finnish language using a causal language modeling (CLM) objective. Llama model was introduced in
|
@@ -106,18 +108,18 @@ The final training dataset had 19 billion words and the evaluation dataset had 2
|
|
106 |
|
107 |
|Dataset | Words | Ratio |
|
108 |
|------------------------------|-------------|-------------|
|
109 |
-
|mc4_3.1.0_fi_cleaned | 11.462B | 60.7
|
110 |
-
|oscar_2301_fi_cleaned | 3.295B | 17.4
|
111 |
-
|Suomi24 | 3.045B | 16.1
|
112 |
-
|multilingual_cc_news | 0.295B | 1.6
|
113 |
-
|STT | 0.249B | 1.3
|
114 |
-
|Yle | 0.201B | 1.1
|
115 |
-
|Reddit_fi_2006_2022 | 0.138B | 0.7
|
116 |
-
|wikipedia_20230501_fi_cleaned | 0.096B | 0.5
|
117 |
-
|Project Lönnrot | 0.078B | 0.4
|
118 |
-
|Finnish parliament speeches | 0.021B | 0.1
|
119 |
-
|fi-news-corpus | 0.004B | 0.1
|
120 |
-
|**TOTAL** | **18.884B** | **100.0
|
121 |
|
122 |
## Training procedure
|
123 |
|
@@ -180,7 +182,7 @@ This model was evaluated using [FIN-bench by TurkuNLP](https://github.com/TurkuN
|
|
180 |
[llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish):
|
181 |
|
182 |
| Task |Version| Metric |Value | |Stderr|
|
183 |
-
|
184 |
|bigbench_analogies | 0|multiple_choice_grade|0.2692|± |0.0391|
|
185 |
|bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.2600|± |0.0441|
|
186 |
|bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.3043|± |0.0981|
|
@@ -229,4 +231,4 @@ This project would not have been possible without compute generously provided by
|
|
229 |
- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
|
230 |
- Rasmus Toivanen, [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)
|
231 |
|
232 |
-
Feel free to contact us for more details 🤗
|
|
|
1 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
2 |
datasets:
|
3 |
- Finnish-NLP/mc4_3.1.0_fi_cleaned
|
4 |
- Finnish-NLP/oscar_2301_fi_cleaned
|
5 |
- Finnish-NLP/Reddit_fi_2006_2022
|
6 |
- Finnish-NLP/wikipedia_20230501_fi_cleaned
|
7 |
- intfloat/multilingual_cc_news
|
8 |
+
language:
|
9 |
+
- fi
|
10 |
+
license: apache-2.0
|
11 |
pipeline_tag: text-generation
|
12 |
+
library_name: transformers
|
13 |
+
tags:
|
14 |
+
- finnish
|
15 |
+
- llama
|
16 |
+
inference: false
|
17 |
---
|
18 |
|
19 |
+
This is the Llama-3b for Finnish model described in the paper [Scaling Data-Constrained Language Models](https://huggingface.co/papers/2305.16264).
|
20 |
+
|
21 |
# Llama-3b for Finnish
|
22 |
|
23 |
Pretrained Llama model on Finnish language using a causal language modeling (CLM) objective. Llama model was introduced in
|
|
|
108 |
|
109 |
|Dataset | Words | Ratio |
|
110 |
|------------------------------|-------------|-------------|
|
111 |
+
|mc4_3.1.0_fi_cleaned | 11.462B | 60.7% |
|
112 |
+
|oscar_2301_fi_cleaned | 3.295B | 17.4% |
|
113 |
+
|Suomi24 | 3.045B | 16.1% |
|
114 |
+
|multilingual_cc_news | 0.295B | 1.6% |
|
115 |
+
|STT | 0.249B | 1.3% |
|
116 |
+
|Yle | 0.201B | 1.1% |
|
117 |
+
|Reddit_fi_2006_2022 | 0.138B | 0.7% |
|
118 |
+
|wikipedia_20230501_fi_cleaned | 0.096B | 0.5% |
|
119 |
+
|Project Lönnrot | 0.078B | 0.4% |
|
120 |
+
|Finnish parliament speeches | 0.021B | 0.1% |
|
121 |
+
|fi-news-corpus | 0.004B | 0.1% |
|
122 |
+
|**TOTAL** | **18.884B** | **100.0%** |
|
123 |
|
124 |
## Training procedure
|
125 |
|
|
|
182 |
[llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish):
|
183 |
|
184 |
| Task |Version| Metric |Value | |Stderr|
|
185 |
+
|------------------------------------------------|------:|---------------------|-----:|---|-----:|\
|
186 |
|bigbench_analogies | 0|multiple_choice_grade|0.2692|± |0.0391|
|
187 |
|bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.2600|± |0.0441|
|
188 |
|bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.3043|± |0.0981|
|
|
|
231 |
- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
|
232 |
- Rasmus Toivanen, [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)
|
233 |
|
234 |
+
Feel free to contact us for more details 🤗
|