 Model Description Polyglot-Ko is a series of large-scale Korean autoregressive language models made by the EleutherAI polyglot team.  | Hyperparameter       | Value                                                                                                                                  | |----------------------|----------------------------------------------------------------------------------------------------------------------------------------| | \\(n_{parameters}\\) | 1,331,810,304                                                                                                                           | | \\(n_{layers}\\)     | 24                                                                                                                                     | | \\(d_{model}\\)      | 2,048                                                                                                                                   | | \\(d_{ff}\\)         | 8,192                                                                                                                                   | | \\(n_{heads}\\)      | 16                                                                                                                                     | | \\(d_{head}\\)       | 128                                                                                                                                    | | \\(n_{ctx}\\)        | 2,048                                                                                                                                   | | \\(n_{vocab}\\)      | 30,003 / 30,080                                                                                                                        | | Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864)                                                                   | | RoPE Dimensions      | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py