readme: ---
  language:
  - en
  - code
  
  
  tags:
  - pytorch
  - causal-lm
  - code-generation
  - The Pile
  
  
  license: apache-2.0
  
  ---
  
  
  # FIM-1.3B 
  
  ## Model Description
  
  FIM-1.3B is the first of a series of large-scale infilling-enabled autoregressive language models trained by CarperAI. FIM-1.3B is the first of these models, and future models (both larger and smaller) trained on greater quantities of code data will be released, potentially with different architectural variations optimized for code.
  
  This is a preliminary release of an experimental artifact and should be treated as such. We are releasing these results and this model in the hopes that it may be useful to the greater research community, especially those interested in LMs for code and pair programming tools.
  
  CarperAI will be releasing larger LMs better tuned for code in the near future, building on these experiments.
  
  
  
  ## Model Dimensions
  
  
  | Hyperparameter       | Value                                                                                                                                  |
  |----------------------|----------------------------------------------------------------------------------------------------------------------------------------|
  | \\(n_{parameters}\\) | 1,331,810,304                                                                                                                           |
  | \\(n_{layers}\\)     | 24                                                                                                                                     |
  | \\(d_{model}\\)      | 2048                                                                                                                                   |
  | \\(d_{ff}\\)         | 8192                                                                                                                                   |
  | \\(n_{heads}\\)      | 16                                                                                                                                     |
  | \\(d_{head}\\)       | 128                                                                                                                                    |
  | \\(n_{ctx}\\)        | 2048                                                                                                                                   |
  | \\(n_{vocab}\\)      | 50280                                                                                                                     |
  | Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864)                                                                   
  
  
  
  The model consists of 24 transformer layers with a hidden dimension of 2048, and a feedforward intermediate dimension of 8192. The hidden dimension is split into 16 heads for self-attention, each with a dimension of 128. Rotary Position Embedding (RoPE) is used.
  
  
  The model is trained with the same tokenizer as [GPT-NeoX-20b](https://arxiv.org/abs/2204.06745), for a vocabulary size of 50254 tokens.
  
  
  ## Training Data
  
  The model was trained on the Pile, an 800Gb dataset composed of varied web corpora. The datasheet and paper for the Pile can be found [here](https://arxiv.org/abs/2201.07311) and [here](https://arxiv.org/abs/2101.00027) respectively.
  
  
  ## Training Details
  
  This model was trained for 47,000 steps at a batch size of 6,291,456 tokens per step in the [GPT-NeoX codebase](https://github.com/EleutherAI/gpt-neox). It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token correctly. 
  
  Following [Bavarian et al. 2022](https://arxiv.org/abs/2207.14255), we train the model to additionally perform infilling via a data transformation applied randomly to 90% of input contexts at train-time. 
  
  Middle segments “to infill” were selected uniformly at random from contexts at the character level, and these contexts were then reformatted as 
  
  \<SUF\> {last 1/3rd of the context} \<PRE\> {first 1/3rd of the context} \<MID\> {middle 1/3rd of the context} \<EOD\>
  
  
  
  
  
  
  ## How to use
  
  
  This model can be easily loaded using the `AutoModelForCausalLM` class:
  
  
  ```python
  
  
  from transformers import AutoTokenizer, AutoModelForCausalLM
  
  
  tokenizer = AutoTokenizer.from_pretrained("CarperAI/FIM-NeoX-1.3B")
  model = AutoModelForCausalLM.from_pretrained("CarperAI/FIM-NeoX-1.3B")
  
  
  ```
  
  ### Performing Infilling
  
  Suppose we have some text that we would like to perform infilling on at a certain “cursor location”. 
  
  This would have the form {some prelude text here} \<INFILLING LOCATION\> {some text following cursor}.
  
  The way to perform infilling generation would be via placing the input text into this format:
  
  \<SUF\> {some text following cursor} \<PRE\> {some prelude text here} \<MID\> ... 
  
  
  language model output is generated after \<MID\> token!
  
  
  As a concrete example, here is a code snippet that should allow a model to perform infilling