File size: 394 Bytes
b3bf026 |
1 |
In this release, the ALIA-40B checkpoint has completed the main pre-training on 8.56 trillion tokens (1.6 epochs with 2.4 trillion tokens and 2 epochs with 2.68 trillion tokens) using a 4k context window, which was unfinished in the previous release. Additionally, it has undergone a preliminary final pre-training stage with a subset of high-quality data and a context extension to 32k tokens. |