readme: ---
  language:
  - sk
  tags:
  - Slovak GPT-J
  - pytorch
  - causal-lm
  license: gpl-3.0
  ---
  
  # Slovak GPT-J-1.4B
  Slovak GPT-J-1.4B with the whopping `1,415,283,792` parameters is the latest and the largest model released in Slovak GPT-J series. Smaller variants, [Slovak GPT-J-405M](https://huggingface.co/Milos/slovak-gpt-j-405M) and [Slovak GPT-J-162M](https://huggingface.co/Milos/slovak-gpt-j-162M), are still available.
  ## Model Description
  Model is based on [GPT-J](https://github.com/kingoflolz/mesh-transformer-jax/) and has over 1.4B trainable parameters.
  
  <figure>
  
  | Hyperparameter       | Value                                                                                                                                  |
  |----------------------|----------------------------------------------------------------------------------------------------------------------------------------|
  | \\(n_{parameters}\\) | 1,415,283,792                                                                                                                          |
  | \\(n_{layers}\\)     | 24                                                                                                                                     |
  | \\(d_{model}\\)      | 2048                                                                                                                                   |
  | \\(d_{ff}\\)         | 16384                                                                                                                                  |
  | \\(n_{heads}\\)      | 16                                                                                                                                     |
  | \\(d_{head}\\)       | 256                                                                                                                                    |
  | \\(n_{ctx}\\)        | 2048                                                                                                                                   |
  | \\(n_{vocab}\\)      | 50256 (same tokenizer as GPT-2/3&dagger;)                                                                                              |
  | Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864)                                                                   |
  | RoPE Dimensions      | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py#L223) |
  
  <p><strong>&dagger;</strong> ByteLevelBPETokenizer was trained on the same Slovak corpus.</p></figure>
  
  ## Training data
  
  Slovak GPT-J models were trained on a privately collected dataset consisting of predominantly Slovak text spanning different categories, e.g. web, news articles or even biblical texts - in total, over 40GB of text data was used to train this model.
  The dataset was preprocessed and cleaned in a specific way that involves minor but a few caveats, so in order to achieve the expected performance, feel free to refer to [How to use] section. Please, keep in mind that despite the effort to remove inappropriate corpus, the model still might generate sensitive content or leak sensitive information.
  
  ## Training procedure
  
  This model was trained for a bit more than 26.5 billion tokens over 48,001 steps on TPU v3-8 pod. The cross-entropy validation loss at the last step was `2.657`.
  
  ## Intended Use
  
  Same as the original GPT-J, Slovak GPT-J learns an inner representation of the language that can be used to extract features useful for downstream tasks, however, the intended use is text generation from a prompt.
  
  ### How to use
  
  This model along with the tokenizer can be easily loaded using the `AutoModelForCausalLM` functionality:
  
  ```python
  from transformers import AutoTokenizer, AutoModelForCausalLM
  
  tokenizer = AutoTokenizer.from_pretrained("Milos/slovak-gpt-j-1.4B")
  model = AutoModelForCausalLM.from_pretrained("Milos/slovak-gpt-j-1.4B")
  ```
  
  When generating a prompt keep in mind these three things, and you should be good to go:
  1. Never leave trailing whitespaces. There's a difference between how tokenizer encodes "Mám rád slovenčinu" (no space after `slovenčinu`) and "Mám rád slovenčinu " (trailing space after `slovenčinu`), i.e `[12805, 2872, 46878]` != `[12805, 2872, 46878, 221]`.
  2. Always use good ol' US English primary double quotation marks, i.e. `""` instead of `„“`.
  3. In case of a new line always enter `\n\n` instead of a single `\n`
  
  To illustrate an example of a basic text generation:
  ```
  >>> prompt = "Tradičné jedlo na Orave sú"
  >>> encoded_input = tokenizer(prompt, return_tensors='pt')
  >>> output = model.generate(**encoded_input)
  >>> tokenizer.decode(output[0])
  'Tradičné jedlo na Orave sú bryndzové halušky\n\nNa Orave sa v minulosti varilo viac druhov'
  ```
  
  ### Capabilities, Limitations, and Biases
  
  Based on the parameter count and our internal testing this should be by far the most capable model out of all models rele