Solved by BramVanroy in post #2 
              
              
                GPT-J has been merged and is part of version 4.11, so you should be able to update your transformers version to the latest one and use GPT-J.