Solved by sgugger in post #2 
              
              
                You should either: 

use the regular torch.load to load the weights of your models
or make sure your custom model class subclasses PreTrainedModel and is initalized with a single config, like all Transformers models if you want to use from_pretrained.