could you add an option to transfer variables from float32 to float16 in GPT2 model to reduce model size and accelerate the inference speed 
## 🚀 Feature
could you add an option to transfer variables from float32 to float16 in GPT2 model
## Motivation
reduce model size and accelerate the inference speed
## Additional context

<!-- Add any other context or screenshots about the feature request here. -->
You should give a look at NVIDIA's apex library and PyTorch `model.half()` method.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@thomwolf Do you know if the gpt2 model needs to be pre-trained with apex support in order to use NVIDIA's apex library (e.g. O1 mode) at inference time? Was mixed precision used during the training of the gpt2 model? Is there a way I can verify that?<|||||>Do we have models trained with mixed precision enabled for gpt2? I can't find them in huggingface's repo. 