Recommended multilingual bert cased model returns similar embeddings 
I'm trying to get embeddings for multilingual input: 
```
tokenizer = BertTokenizer.from_pretrained("bert-base-multilingual-cased", do_lower_case=False)

class NeuralNet(BertPreTrainedModel):
    def __init__(self, config):
        super(NeuralNet, self).__init__(config)
        self.bert = BertModel(config)
        self.apply(self.init_bert_weights)
    
    def forward(self, input_ids, token_type_ids=None, attention_mask=None):
        _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)

        return pooled_output

model = NeuralNet.from_pretrained("bert-base-multilingual-cased")
```

and for some reason all `pooled_output` vectors are very similar with 1e-3 cosine distance for semantically different inputs. Changing model to `bert-base-multilingual-UNcased` works just okay. Any ideas?
I second this issue #735 <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
