Roberta embeddings comparison
## ❓ Questions & Help
**SYSTEM**
OS: Linux pop-os 5.0.0
Python version: 3.6.8
Torch version: 1.3.0
Transformers version: 2.1.1
I am running this linux VM with the above software versions on a Windows 10 laptop.

I wanted to compare the cosine similarity between elmo and roberta embeddings for two sequences:
```python
from allennlp.commands.elmo import ElmoEmbedder
from allennlp.data.tokenizers.spacy_tokenizer import SpacyTokenizer
from scipy.spatial.distance import cdist
import numpy as np
import pandas as pd
from transformers import RobertaTokenizer, RobertaModel
import torch

seqA = 'How many cars are in the target parking lot'
seqB = 'How many countries are in the continent of Africa'
comparison_df = pd.DataFrame()
sim_list = []

model_types = ['elmo', 'roberta']
for model_type in model_types:
    sim_list = []
    if model_type == 'elmo':
        tokenizer = SpacyTokenizer()
        model = ElmoEmbedder()
        seqA_tokens = np.array([s.text for s in tokenizer.tokenize(seqA)])
        seqB_tokens = np.array([s.text for s in tokenizer.tokenize(seqB)])

        seqA_embeddings = model.embed_sentence(seqA_tokens)
        seqB_embeddings = model.embed_sentence(seqB_tokens)

        seqA_embeds = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))
        seqB_embeds = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))
        Y = cdist(seqA_embeds, seqB_embeds, 'cosine')[0][0]
        cos_sim = 1.0 - Y
        sim_list.append(cos_sim)


    elif model_type == 'roberta':
        tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
        model = RobertaModel.from_pretrained('roberta-large')
        seqA_tokens = torch.tensor(tokenizer.encode(seqA, add_special_tokens=True)).unsqueeze(0)
        seqB_tokens = torch.tensor(tokenizer.encode(seqB, add_special_tokens=True)).unsqueeze(0)

        with torch.no_grad():
            seqA_embeddings = model(seqA_tokens)[0]
            seqB_embeddings = model(seqB_tokens)[0]

        seqA_embeddings = seqA_embeddings.detach().numpy()
        seqA_embeddings = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))

        seqB_embeddings = seqB_embeddings.detach().numpy()
        seqB_embeddings = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))

        Y = cdist(seqA_embeddings, seqB_embeddings, 'cosine')[0][0]
        cos_sim = 1.0 - Y
        sim_list.append(cos_sim)

    comparison_df[model_type] = sim_list
print(comparison_df)
```

The result of the print statement is:
```
      elmo   roberta
0  0.55092  0.996526
```

I find this strange because, despite having the same number of tokens, I would expect the similarity between the two sequences to not be so close to 1 as roberta is showing. My concern is that I am not accessing the embeddings correctly or not pooling correctly (or both), and was wondering if people here wouldn't mind letting me know what I am doing wrong? Thanks in advance!

Any update on this issue?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> ## ❓ Questions & Help
> **SYSTEM**
> OS: Linux pop-os 5.0.0
> Python version: 3.6.8
> Torch version: 1.3.0
> Transformers version: 2.1.1
> I am running this linux VM with the above software versions on a Windows 10 laptop.
> 
> I wanted to compare the cosine similarity between elmo and roberta embeddings for two sequences:
> 
> ```python
> from allennlp.commands.elmo import ElmoEmbedder
> from allennlp.data.tokenizers.spacy_tokenizer import SpacyTokenizer
> from scipy.spatial.distance import cdist
> import numpy as np
> import pandas as pd
> from transformers import RobertaTokenizer, RobertaModel
> import torch
> 
> seqA = 'How many cars are in the target parking lot'
> seqB = 'How many countries are in the continent of Africa'
> comparison_df = pd.DataFrame()
> sim_list = []
> 
> model_types = ['elmo', 'roberta']
> for model_type in model_types:
>     sim_list = []
>     if model_type == 'elmo':
>         tokenizer = SpacyTokenizer()
>         model = ElmoEmbedder()
>         seqA_tokens = np.array([s.text for s in tokenizer.tokenize(seqA)])
>         seqB_tokens = np.array([s.text for s in tokenizer.tokenize(seqB)])
> 
>         seqA_embeddings = model.embed_sentence(seqA_tokens)
>         seqB_embeddings = model.embed_sentence(seqB_tokens)
> 
>         seqA_embeds = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))
>         seqB_embeds = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))
>         Y = cdist(seqA_embeds, seqB_embeds, 'cosine')[0][0]
>         cos_sim = 1.0 - Y
>         sim_list.append(cos_sim)
> 
> 
>     elif model_type == 'roberta':
>         tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
>         model = RobertaModel.from_pretrained('roberta-large')
>         seqA_tokens = torch.tensor(tokenizer.encode(seqA, add_special_tokens=True)).unsqueeze(0)
>         seqB_tokens = torch.tensor(tokenizer.encode(seqB, add_special_tokens=True)).unsqueeze(0)
> 
>         with torch.no_grad():
>             seqA_embeddings = model(seqA_tokens)[0]
>             seqB_embeddings = model(seqB_tokens)[0]
> 
>         seqA_embeddings = seqA_embeddings.detach().numpy()
>         seqA_embeddings = np.sum(seqA_embeddings[-1], axis=0).reshape((1, seqA_embeddings.shape[2]))
> 
>         seqB_embeddings = seqB_embeddings.detach().numpy()
>         seqB_embeddings = np.sum(seqB_embeddings[-1], axis=0).reshape((1, seqB_embeddings.shape[2]))
> 
>         Y = cdist(seqA_embeddings, seqB_embeddings, 'cosine')[0][0]
>         cos_sim = 1.0 - Y
>         sim_list.append(cos_sim)
> 
>     comparison_df[model_type] = sim_list
> print(comparison_df)
> ```
> 
> The result of the print statement is:
> 
> ```
>       elmo   roberta
> 0  0.55092  0.996526
> ```
> 
> I find this strange because, despite having the same number of tokens, I would expect the similarity between the two sequences to not be so close to 1 as roberta is showing. My concern is that I am not accessing the embeddings correctly or not pooling correctly (or both), and was wondering if people here wouldn't mind letting me know what I am doing wrong? Thanks in advance!

@aclifton314 have you deduce any meaningful intuition on this thus far ?? I found another similar issue, hopefully this can help: https://github.com/huggingface/transformers/issues/2298 ...<|||||>@aclifton314 

In addition to the above link, I also ran a quick test with most popular models not just Roberta, it seems there are some models that can differentiate better, though I am not sure if you can generalize that statement without the test. Code and test results below:

```
import torch
from torch import nn
from transformers import *

import numpy as np
from scipy.spatial import distance
from sklearn.metrics.pairwise import cosine_similarity

# for 10 transformer architectures and 30 pretrained weights.
#          Model          | Tokenizer          | Pretrained weights shortcut
MODELS = [(BertModel,       BertTokenizer,       'bert-base-uncased'),
          (OpenAIGPTModel,  OpenAIGPTTokenizer,  'openai-gpt'),
          (GPT2Model,       GPT2Tokenizer,       'gpt2'),
          (CTRLModel,       CTRLTokenizer,       'ctrl'),
          (TransfoXLModel,  TransfoXLTokenizer,  'transfo-xl-wt103'),
          (XLNetModel,      XLNetTokenizer,      'xlnet-base-cased'),
          (XLMModel,        XLMTokenizer,        'xlm-mlm-enfr-1024'),
          (DistilBertModel, DistilBertTokenizer, 'distilbert-base-uncased'),
          (RobertaModel,    RobertaTokenizer,    'roberta-base'),
          (XLMRobertaModel, XLMRobertaTokenizer, 'xlm-roberta-base'),
         ]

for model_class, tokenizer_class, pretrained_weights in MODELS:
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights)

    print ("")
    print ("Model", ''.join(e for e in str(type(model)).split(".")[-1] if e.isalnum()))

    input_ids1 = torch.tensor([tokenizer.encode("How many cars are in the target parking lot", add_special_tokens=True)])                                            
    input_ids2 = torch.tensor([tokenizer.encode("How many countries are in the continent of Africa", add_special_tokens=True)])  

    with torch.no_grad():
        last_hidden_states1 = model(input_ids1)[0].detach().numpy()
        last_hidden_states2 = model(input_ids2)[0].detach().numpy()
        
        last_hidden_states1_sq = np.sum(last_hidden_states1[-1], axis=0).reshape((1, last_hidden_states1.shape[2]))
        last_hidden_states2_sq = np.sum(last_hidden_states2[-1], axis=0).reshape((1, last_hidden_states2.shape[2]))
        
        print ("Similarity between Sent1 & Sent 2", cosine_similarity(last_hidden_states1_sq, last_hidden_states2_sq) )

```
**Results**:

Model BertModel
Similarity between Sent1 & Sent 2 [[0.7234119]]

Model OpenAIGPTModel
Similarity between Sent1 & Sent 2 [[0.6779459]]

Model GPT2Model
Similarity between Sent1 & Sent 2 [[0.9986031]]

Model CTRLModel
Similarity between Sent1 & Sent 2 [[0.9086089]]

Model TransfoXLModel
Similarity between Sent1 & Sent 2 [[0.77547705]]

Model XLNetModel
Similarity between Sent1 & Sent 2 [[0.98956037]]

Model XLMModel
Similarity between Sent1 & Sent 2 [[0.623127]]

Model DistilBertModel
Similarity between Sent1 & Sent 2 [[0.7640958]]

Model RobertaModel
Similarity between Sent1 & Sent 2 [[0.96598077]]

Model XLMRobertaModel
Similarity between Sent1 & Sent 2 [[0.99813896]]

**I am interested in Passage (Few Paragraphs) Level Embeddings based Similarity Comparison, if anyone has any viable solution, suggestions please do share.** 

Thanks!!