GPT-2 models are unpickable
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: 

Hi,

I'm trying to train a GPT-2 Double Heads Model (based on your transfer-learning-conv-ai guide) using Pytorch Lightning. However I have a problem when trying to train the model on ddp distributed backend : the `GPT2DoubleHeadsModel` class seems to be unpickable and my training script fails with the following error : 
`TypeError: can't pickle torch._C.ScriptFunction objects`



## To reproduce

Run :
```
from transformers import GPT2DoubleHeadsModel
import pickle
model = GPT2DoubleHeadsModel.from_pretrained("gpt2-medium")
pickle.dump(model,open("test.bin","wb"))
````
The problem does not occur when using `bert-base-uncased` for example. I tried to search which part of GPT-2 class contains ` torch._C.ScriptFunction objects` without success. Do you have an idea to avoid this error ?

Thanks in advance.

- `transformers` version: 2.8
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.5 Cuda 10.2
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes, ddp

Hi! I think this may come from the [activation functions](https://github.com/huggingface/transformers/blob/master/src/transformers/activations.py), some of which are jitted. This allows better performance.

In what situation would you use `pickle` rather than our API `save_pretrained` ?

```py
from transformers import GPT2DoubleHeadsModel
import pickle
model = GPT2DoubleHeadsModel.from_pretrained("gpt2-medium")
model.save_pretrained("test")
```<|||||>Hi Lysandre,

Thanks for your answer !
I use [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) which requires the model to be picklable (https://pytorch-lightning.readthedocs.io/en/stable/multi_gpu.html#make-model-picklable). I see from your link that if the PyTorch version is <1.4, the activation function is not jitted : 
https://github.com/huggingface/transformers/blob/fa49b9afeab5545f14b3661b35195b829fcf8ef5/src/transformers/activations.py#L32-L44

I think i'll test again downgrading my PyTorch version to 1.3.1.<|||||>I understand, this is indeed an issue. We've had other several issues due to this, I think it would be better to revert to picklable option. Will open a PR with this objective in a bit.<|||||>Thank you for your responsiveness !<|||||>It should be fixed on master now. Could you let me know if running the code from the master branch solves your issue?<|||||>Yes, it works, thank you 👌<|||||>I am getting the same error trying to save AlbertForTokenClassification() model using `torch.save(model, 'temp.pt')`
But `torch.save(model.state_dict(), 'temp.pt')` works.<|||||>@paramansh  Do you mind opening an issue with your specific problem, your software versions and code sample so that we may debug on our side? Thanks.