Tokens Jumbling
I have trained a "bert-base-uncased" model (with lesser layers) on my dataset with my tokenizer for sentence similarity task. The final sentence embedding is formed using mean pooling strategy. Now during inference if my tokens for sentence1 is [t1,t2,t3,t4,t5] and for senetence2, I randomly shuffle these tokens example [t3,t1,t2,t5,t4], the score is really high, but sentence2 does't make any sense.
I tested the pretrained bert-base-uncased model and found out same problem, as shown below
Enter text1:a man is riding a horse
Enter text2:a riding man is a horse
Tokens1: [101, 1037, 2158, 2003, 5559, 1037, 3586, 102]
Tokens2: [101, 1037, 5559, 2158, 2003, 1037, 3586, 102]
Similarity: 0.703365683555603
Enter text1:A boy can throw a stone up to a maximum height
Enter text2:A stone up to a boy can maximum throw a height
Tokens1: [101, 1037, 2879, 2064, 5466, 1037, 2962, 2039, 2000, 1037, 4555, 4578, 102]
Tokens2: [101, 1037, 2962, 2039, 2000, 1037, 2879, 2064, 4555, 5466, 1037, 4578, 102]
Similarity: 0.9277969598770142
![aa](https://user-images.githubusercontent.com/47495143/123504257-b3250e80-d675-11eb-90b6-99bf8743e6ad.jpeg)

Which model are you using exactly? `bert-base-uncased` is not a `sentence-transformers` model<|||||>i used "bert-base-uncased"  architecture from here: https://www.sbert.net/docs/training/overview.html to train for sentence embedding tak. I modified the number of layers and hidden size. 
Also as shown in above image, i also tested "bert-base-nli-cls-token" (from https://huggingface.co/sentence-transformers/bert-base-nli-cls-token) and also tested (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens). But all shows same issue
<|||||>Pinging @nreimers <|||||>Hi @PhenomenalOnee 
the two models you linked are deprecated, I recommend to use  the paraphrase v2 models. 

But they will show the same behavior. That is how it is. The models don't check if a sentence makes sense or is grammatically correct. They try to infer the semantics of the sentence and they try to be robust for spelling mistakes, grammatical errors, word shuffleing etc.

If this is undesired for your task, you must create training data that teaches the network that these examples should not be close in the vector space.

<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.