🐛 BART : Same representations for different `<s>` tokens 
# 🐛 Bug

## Context

I'm trying to use BART for a sentence classification task. So I encode the input with the following format :

```
<s> Sen1 </s> <s> Sen2 </s> <s> Sen3 </s> ...
```

And use `<s>` as sentence representation (from the encoder, not decoder). Then I classify these representations.

---

I trained my model but the classification give random choice. After debugging, I noticed that the encoder produce always the same representation for `<s>` token. 

## Bug

**The representation of `<s>` token is always the same, no matter where they appear in the input.**

Here is a notebook reproducing the issue : [Colab](https://colab.research.google.com/drive/1mqKKFAEGEwa5XbkJtm7_VrmQ8L3_0Bnt?usp=sharing)

In this notebook I simply encode an input, modify the input to add an additional `<s>` token, forward it through BART and compare the encoder representation of `<s>`. It gives me :

```
tensor([-0.0097,  0.0075,  0.0086,  ...,  0.0041, -0.0085, -0.0011],
       grad_fn=<SelectBackward>)
tensor([-0.0097,  0.0075,  0.0086,  ...,  0.0041, -0.0085, -0.0011],
       grad_fn=<SelectBackward>)
```

---

Now if I do the same with `</s>` token, their representation is different :

```
tensor([ 0.0658,  0.0161, -0.0062,  ..., -0.0536, -0.0515,  0.1837],
       grad_fn=<SelectBackward>)
tensor([ 0.0627,  0.0576,  0.0408,  ..., -0.0406, -0.0765,  0.1689],
       grad_fn=<SelectBackward>)
```

Which makes sense because they represent different thing...

## Note 1

I think this is a bug because I remember a few months ago, I tried to do the same (use `<s>` representation for classification) with fairseq version of BART, and it worked...

## Note 2

I'm aware the example in the Notebook is using just the pre-trained version of BART, so the `<s>` representation does not represent sentence. But even after training the model, the exact same behavior arise.

@sshleifer 
Very strange indeed. Nothing comes to mind, but if you can show the fairseq discrepancies I can take a look.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@astariul Hi I also encounter this problem. The finetuned BART just gives the same output for `<s>` under eval mode, have you fixed it? <|||||>@turing-yfqiu I'm sorry it's long time ago, I don't remember how I did....