🌟 New model addition - GPT-J-6B
# 🌟 New model addition - GPT-J-6B

## Model description

The GPT-J-6B model (GPT-NEO model in Jax with 6B parameters trained on the Pile)

Repo: https://github.com/kingoflolz/mesh-transformer-jax

Weights:
[Slim weights (bf16 weights only, for inference, 9GB)](https://the-eye.eu/public/AI/GPT-J-6B/step_383500_slim.tar.zstd)
[Full weights (including optimizer params, 61GB)](https://the-eye.eu/public/AI/GPT-J-6B/step_383500.tar.zstd)

## Open source status

* [x] the model implementation is available: (give details)
* [x] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)

Hello @patrickvonplaten! would you be able to give an estimation for the timeline of the implementation of the this model in huggingface? <|||||>I have a PR adding support for this model here: #12098<|||||>> I have a PR adding support for this model here: #12098

You probably wanted to link this PR: https://github.com/huggingface/transformers/pull/12106<|||||>Yeah, copied the wrong thing somehow.<|||||>@finetuneanon great! Do you know when it will be ready to use from the transformer library? Thnx for the work. <|||||>Depends on when it will be merged. Until then you can install my branch like this:

```
pip install git+https://github.com/finetuneanon/transformers@gpt-j
```

Convert the weights with the conversion script linked from the PR.<|||||>@finetuneanon I did pip install the transformers@gpt-j and I managed to convert the weights through the script you referenced but only thing I'm now struggling with is making the config file. I uploaded the gpt-j-6b.json file to colab but I don't how to make the config variable via AutoConfig class(don't know if that is how it is made). So If you could let me know how to make the config file, I would appreciate it a lot. 
this [colab](https://colab.research.google.com/drive/1xl5tRYTiVISn6FMfhgyB70LZ-Biwj43E#scrollTo=QI_zE5QA8ycF) file containts all the code. <|||||>Rename it into config.json, put it into a folder and you should be able to `AutoConfig.from_pretrained("whatever-folder")`