migrate BertForQuestionAnswering from pytorch-pretrained-bert not produce the same result
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): retrained Bert
Language I am using the model on (English, Chinese....): multilingual - vietnamese
The tasks I am working on is:
*  an official GLUE/SQUaD task: SQUaD 
*  my own task or dataset: same format as SQUaD

## To Reproduce

Steps to reproduce the behavior:

I have trained a bert model on older pytorch-pretrained-bert and it works just fine. recently, I switch the code to the latest version of transformer. I use the following config:

>     bert_model = './my_model'
>     max_seq_length = 160
>     doc_stride = 160
>     predict_batch_size = 20
>     n_best_size=20
>     max_answer_length=30
>     verbose_logging = False
>     no_cuda = True
>     seed= 42
>     do_lower_case= True
>     version_2_with_negative = True
>     null_score_diff_threshold=0.0
>     max_query_length = 64
>     THRESH_HOLD = 0.95

I import 2 class:
`from transformers import BertForQuestionAnswering as bqa1`
`from pytorch_pretrained_bert.modeling import BertForQuestionAnswering as bqa2`
and load 2 model as following :
`model1 = bqa1.from_pretrained(args.bert_model)`
`model2 = bqa2.from_pretrained(args.bert_model)`

and input to models with the same tensors:
`input_ids = torch.ones((1,160),dtype = torch.int64)`
`segment_ids = torch.ones((1,160),dtype = torch.int64)`
`input_mask = torch.ones((1,160),dtype = torch.int64) `
`model(input_ids, segment_ids, input_mask)`

I also check if 2 model has same weights or not by using following guide [https://discuss.pytorch.org/t/check-if-models-have-same-weights/4351/3](guide).

I seed the randomness of torch before inference 2 model by using:
`seed = 0`
`torch.manual_seed(seed)`
`if torch.cuda.is_available():`
`    torch.cuda.manual_seed_all(seed)`
but 2 models still produce difference results.
Hello! Have you put these models in `eval()` mode so as to deactivate the dropout modules?<|||||>For completeness sake: did you train both models with the same random seed? Or are you just trying to evaluate models that you trained?

My go-to method is:

```python
def set_seed(seed):
    """ Set all seeds to make results reproducible (deterministic mode).
        When seed is a false-y value or not supplied, disables deterministic mode. """

    if seed:
        logging.info(f"Running in deterministic mode with seed {seed}")
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
        np.random.seed(seed)
        random.seed(seed)
        os.environ['PYTHONHASHSEED'] = str(seed)
    else:
        logging.info(f"Running in non-deterministic mode")
```<|||||>It does work when I put generated tensors into 2 models but doesn't when I put tensors I save before. Maybe I will rewrite the inference code with new code base and retrain the model.<|||||>Are you sure your model is in evaluation mode? <|||||>yes, I already put it in evaluation mode<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
