Update README.md

README.md

@@ -26,6 +26,15 @@ Training was conducted on the [LUMI supercomputer](https://www.lumi-supercompute
 The project aimed to train multilingual encoder models that support long context and all official Finnish languages¹. The model can theoretically extrapolate to a context length of 128,000 tokens.
 
 ¹Multiple Sámi languages are spoken in Finland, but Northern Sámi is the most widespread and thus included in the training data. English is not the official language of Finland, but it is widely used. Latin was included for potential clinical use.
+## Table of Contents
+1. [Model Overview](#model-overview)
+2. [Training](#training)
+3. [Training data](#training-data)
+4. [Evaluation results](#evaluation-results)
+5. [Ethical Considerations and Limitations](#ethical-considerations-and-limitations)
+6. [Aknowledgements](#aknowledgements)
+7. [Licence](#licence)
+8. [Citation information](#citation-information)
 ## Model Overview
 | Hyperparameter | Value  |
 | :------------- | :----: |
@@ -36,9 +45,9 @@ The project aimed to train multilingual encoder models that support long context
 | sequence_length | 16,000 / 128,000 |
 ## Training
 Pretraining was done using Distributed Data Parallelism, AdamW with ZeroRedundancyOptimizer, and the WSD learning rate schedule. 
-The model was trained with a learning rate of 5e-4, a sequence length of 1024, and a RoPE theta of 10,000 for 377B tokens over 117,300 steps.
+The model was trained with a learning rate of 8e-4, a sequence length of 1024, and a RoPE theta of 10,000 for 377B tokens over 117,300 steps.
 ### Long context training
-The model was trained with a learning rate of 3e-4, increasing the context length from 1024 to 16,000 in six stages, 
+The model was trained with a learning rate of 5e-4, increasing the context length from 1024 to 16,000 in six stages, 
 where each sequence length was trained for an equal number of tokens, totaling 53B tokens over 16,560 steps.
 RoPE theta in global layers was increased to 1,000,000. Long documents were sampled from the original data in the distribution below:
 |Sequence lenght | % |