BERT outputs are different with the same input in training mode
When the training mode is enabled, BERT model returns different outputs even for the same input, is there any idea on why this happens?

```python
from transformers import BertModel, BertTokenizer
model = BertModel.from_pretrained("bert-base-uncased")
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
a = tokenizer.encode("Hello how are you?", return_tebsors='pt')
model.train()

torch.mean(model(a)[0])
# tensor(-0.0167, grad_fn=<MeanBackward0>)
torch.mean(model(a)[0])
# tensor(-0.0162, grad_fn=<MeanBackward0>)
torch.mean(model(a)[0])
# tensor(-0.0156, grad_fn=<MeanBackward0>)


```
Hi there! The [forum](https://discuss.huggingface.co/) is a better place for those kinds of general questions, as we keep the issues for bugs and feature requests only.

To answer your question, this is because most Deep Learning models (including BERT) use a technique called [dropout](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) to generalize better, which randomly zeros some activations during training. This randomness is the reason you are getting different results for the same inputs.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @lamthuy were you able to investigate and find out the cause?<|||||>Same question here!<|||||>I am also facing the same error. 
@sgugger I thought the dropout is deactivated once I call model.eval()

<|||||>Not sure what your code sample is @lava18 . The code sample above (fixed like below) always returns the same value once the line `model.train()` is removed.
```py
import torch
from transformers import BertModel, BertTokenizer
model = BertModel.from_pretrained("bert-base-uncased")
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
a = tokenizer.encode("Hello how are you?", return_tensors='pt')

torch.mean(model(a).to_tuple()[0])
```<|||||>You need to call `model.eval()` after training (or before inference). That should deactivate the dropouts and you will always get the same output for the same input. 