Support fp16 for inference
# 🚀 Feature request - support fp16 inference

Right now most models support mixed precision for model training, but not for inference. Naively calling `model= model.haf()` makes the model generate junk instead of valid results for text generation, even though mixed-precision works fine in training. 

If there's a way to make the model produce stable behavior at 16-bit precision at inference, the throughput can potentially double on most modern GPUS. 

## Motivation

Double the speed is always attractive, especially since transformers are compute-intensive. 


I imagine this only happens when the `generate()` method is used under the hood? <|||||>Hi there!  It's true you can't just do `model.half()` for generation. There is nothing in Trainer/Seq2SeqTrainer right now for FP16-inference, only training, but we're looking at it right now through #8403. So stay tuned!<|||||>Thanks for the input @sgugger. Good to know we're not missing something here and it's actually unsupported somehow. <|||||>Hi,
I've noticed the same issue of the model randomly generating junk when using autocast within a custom generate() method with the only change below (fp16 is a boolean). From the above comments I thought this approach should've worked.
```
if fp16:
	with torch.cuda.amp.autocast():
		outputs = self(**model_inputs)
else:
	outputs = self(**model_inputs)
```

The current model I've tested it on is a huggingface gpt2 model finetuned on a personal dataset. Without fp16 the generate works perfectly. The dataset is very specific and the model is supposed to generate symbols+numbers, so it's clear when it starts spitting out words during fp16 inference.
<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.<|||||>has this been solved?