Errors when using Apex
Using `pytorch-pretrained-BERT` with `apex` installed breaks with the errors below. I am using it through allennlp, and my environment settings are: 
```
ubuntu 16.04
nvidia driver 410.48
4 Titan V gpus
python 3.6.8
cuda 9
pytorch 1.0.1.post2
pytorch-pretrained-bert 0.6.1 
```
I also tried python 3.7, cuda 10 and nvidia driver 390 with the same errors.

At training time, the crashes from the first batch with the following error:
```
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/allennlp/modules/text_field_embedders/basic_text_field_embedder.py", line 110, in forward
    token_vectors = embedder(*tensors)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/allennlp/modules/token_embedders/bert_token_embedder.py", line 91, in forward
    attention_mask=util.combine_initial_dims(input_mask))
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 711, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 263, in forward
    embeddings = self.dropout(embeddings)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/modules/dropout.py", line 58, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/home/beltagy/miniconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 749, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /opt/conda/conda-bld/pytorch_1549636813070/work/aten/src/THC/THCGeneral.cpp:405
```

At prediction time, it crashes with the following error after exactly 13 batches (even when I shuffle the data or change the batch size) 
```  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/allennlp/modules/text_field_embedders/basic_text_field_embedder.py", line 110, in forward
    token_vectors = embedder(*tensors)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/allennlp/modules/token_embedders/bert_token_embedder.py", line 91, in forward
    attention_mask=util.combine_initial_dims(input_mask))
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 714, in forward
    output_all_encoded_layers=output_all_encoded_layers)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 396, in forward
    hidden_states = layer_module(hidden_states, attention_mask)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 381, in forward
    attention_output = self.attention(hidden_states, attention_mask)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 339, in forward
    self_output = self.self(input_tensor, attention_mask)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 290, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 67, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/beltagy/miniconda3/envs/cuda9/lib/python3.6/site-packages/torch/nn/functional.py", line 1354, in linear
    output = input.matmul(weight.t())
RuntimeError: cublas runtime error : an internal operation failed at /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCBlas.cu:258
```

Any thoughts what might be wrong, or how I can debug this ? 
Can you try to run it with `CUDA_LAUNCH_BLOCKING=1` so we can see which exact CUDA call fails?
Also, do you have a simple way for me to try to reproduce this error?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
