BertModel sometimes produces the same output during evaluation
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I finetune BertModel as a part of my model to produce word embeddings. I found sometimes the performance was very bad, then i run the code again without any change, the performance was normal. It is very strange. I check my code to try to find the bug. I found the word embeddings produced by BertModel were all the same. Then I followed the code of BertModel, and found the BertEncoder would make the output become similar gradually which was consisted of 12 BertLayers. I have no idea about this situation.
Did you set a fixed seed? If you want deterministic results, you should set a fixed seed. <|||||>> Did you set a fixed seed? If you want deterministic results, you should set a fixed seed.

No, I didn't. My problem seems that the model finetuned is totally wrong sometimes. Is it a problem related to random seed?<|||||>What do you mean by "bad"? Post some code to better help you.<|||||>> What do you mean by "bad"? Post some code to better help you.

For example, the normal performance is 140.1, but the bad performance is 1.0. As I mentioned before, this situation happened sometimes and I found the word embeddings produced by BertModel were all the same when the performance was bad.

The text encoding part of my code is as follows:
 
![image](https://user-images.githubusercontent.com/37296256/71330710-697fd300-2569-11ea-8839-583502b8bcdc.png)
<|||||>I think you are accessing the CLS token embeddings and It is constant as the model you are using is trained on MLM objective. <|||||>> I think you are accessing the CLS token embeddings and It is constant as the model you are using is trained on MLM objective.

If I change the out to self.bert(text)[0][:,1,:], the output is the same as self.bert(text)[0][:,0,:]. It seems I get the same output no matter the input that I put in.<|||||>I saved the checkpoint when this situation happened. I multiplied all the parameter values of bert by 10 and found the outputs were different, while I divided all parameter values by 10 and found the outputs were almost same. So I think the reason is that the parameter values are too small.<|||||>@xuesong0309  https://github.com/huggingface/transformers/issues/1465，Could you look at this problem?<|||||>> @xuesong0309  https://github.com/huggingface/transformers/issues/1465，Could you look at this problem?

Did you always get same output? I suggest outputing the parameter values of your model as I mentioned above.<|||||>> > @xuesong0309  [https://github.com/huggingface/transformers/issues/1465，Could](https://github.com/huggingface/transformers/issues/1465%EF%BC%8CCould) you look at this problem?
> 
> Did you always get same output? I suggest outputing the parameter values of your model as I mentioned above.

It may not be the problem you mentioned, because the model is normal for multi-class classification, and this happens only for multi-label classification.<|||||>> @xuesong0309  https://github.com/huggingface/transformers/issues/1465，Could you look at this problem?
> 
> Did you always get same output? I suggest outputing the parameter values of your model as I mentioned above.
> 
> It may not be the problem you mentioned, because the model is normal for multi-class classification, and this happens only for multi-label classification.

You could open a new issue to describe your problem in detail.<|||||>Have you solved this issue? I have facing the same issue. Output same results during evaluation.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
