gpt2.generate breaks on FP16 Apex training.
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): Molecule

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Run Pytorch lightning with Apex (or just default apex training), then during validation try and generate samples with the model (which is on fp16).

## Trace

```
2020-06-23T18:30:00.337+01:00 | THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCReduceAll.cuh line=327 error=716 : misaligned address
-- | --
  | 2020-06-23T18:30:00.338+01:00 | Validation sanity check: 0it [00:00, ?it/s] Validation sanity check: 50% 1/2 [00:01<00:01, 1.08s/it]generating smiles
  | 2020-06-23T18:30:00.341+01:00 | Traceback (most recent call last):
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/bin/transformervae", line 11, in <module>
  | 2020-06-23T18:30:00.341+01:00 | load_entry_point('exs-transformervae', 'console_scripts', 'transformervae')()
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  | 2020-06-23T18:30:00.341+01:00 | return self.main(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 782, in main
  | 2020-06-23T18:30:00.341+01:00 | rv = self.invoke(ctx)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return _process_result(sub_ctx.command.invoke(sub_ctx))
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return ctx.invoke(self.callback, **ctx.params)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return callback(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/cli.py", line 404, in pretrain
  | 2020-06-23T18:30:00.341+01:00 | trainer.fit(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/uwandb: Waiting for W&B process to finish, PID 28
  | 2020-06-23T18:30:00.341+01:00 | ser/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in fit
  | 2020-06-23T18:30:00.341+01:00 | self.single_gpu_train(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 176, in single_gpu_train
  | 2020-06-23T18:30:00.341+01:00 | self.run_pretrain_routine(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1076, in run_pretrain_routine
  | 2020-06-23T18:30:00.341+01:00 | False)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 330, in _evaluate
  | 2020-06-23T18:30:00.341+01:00 | eval_results = model.validation_epoch_end(outputs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/base.py", line 95, in validation_epoch_end
  | 2020-06-23T18:30:00.341+01:00 | return self._shared_eval_end(output, "val")
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/lm.py", line 152, in _shared_eval_end
  | 2020-06-23T18:30:00.341+01:00 | use_cache=True,
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | return func(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/lm.py", line 173, in generate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | generated_ids = self.encoder.generate(input_ids, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | return func(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1181, in generate
  | 2020-06-23T18:30:00.341+01:00 | model_specific_kwargs=model_specific_kwargs,
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1285, in _generate_no_beam_search
  | 2020-06-23T18:30:00.341+01:00 | if unfinished_sents.max() == 0:
  | 2020-06-23T18:30:00.341+01:00 | RuntimeError: cuda runtime error (716) : misaligned address at /opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCReduceAll.cuh:327


```

## Expected behavior

Should generate samples.


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 cu101
- Tensorflow version (GPU?): n/a
- Using GPU in script?: V100
- Using distributed or parallel set-up in script?: no but FP16 training

This may be an error on my part, apologies. Just confirming.<|||||>This is supposed to work, misaligned address is usually a flavor of OOM. I would try to cut your batch size.<|||||>@sshleifer Batch size is 16, am on a V100 on FP16. Also I'm training with a batch size of 512, so surely batch size is fine in this case?<|||||>Okay, turns out the problem is still there.

Here is my code:

```
    @torch.no_grad()
    def generate_no_grad(self, num_samples, batch_size, **kwargs):
        device = next(self.parameters()).device
        num_iters = num_samples // batch_size
        all_smiles = []
        for idx in range(num_iters):
            input_ids = torch.empty(batch_size, 1).fill_(self.tokenizer.bos_token_id)
            input_ids = input_ids.to(device).long()
            generated_ids = self.encoder.generate(input_ids, **kwargs)
            smiles = self.tokenizer.decode(generated_ids)
            all_smiles.extend(smiles)
        return all_smiles

    def generate(self):
            smiles = self.generate_no_grad(
                num_samples=50,
                batch_size=16,
                max_length=self.collater.max_length,
                do_sample=True,
                num_beams=1,
                temperature=1.0,
                top_k=500,
                top_p=1.0,
                repetition_penalty=1.0,
                pad_token_id=self.tokenizer.pad_token_id,
                bos_token_id=self.tokenizer.bos_token_id,
                eos_token_id=self.tokenizer.eos_token_id,
                length_penalty=1,
                no_repeat_ngram_size=0,
                num_return_sequences=1,
                use_cache=True,
            )
            ....
```

Bugs out on batch size 16, num_samples=10_000, max_length=100<|||||>@patrickvonplaten  Any idea with this?

Also, I'm on torch 1.4 and cu101, could upgrading to 1.5.1 and cu102 fix this?<|||||>I cannot reproduce the error in the notebook. Looking into it more. <|||||>Hmmmm, getting the same error on batch size 8 and num samples = 50.

here's my colab where I'm trying to reproduce: https://colab.research.google.com/drive/13uvd_Y_VHoZqQxyZ0OdyyNAJEurrW2LX?usp=sharing

@sshleifer I'm guessing the CICD checks for torch 1.4 cu101?<|||||>Okay. I can only think that this is a pytorch 1.4 cu101 error. Will update image to 1.5.1, try to tomorrow and update this accordingly. Cheers!<|||||>Is there anything from the code above that would warrant this error?<|||||>Well, well, well @sshleifer I moved to the pytorch nightly conda builds (1.6-dev, which has native AMP) and now the issue is no longer there.

Weird ...<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
