CANINE model gets different logits for different batch sizes
## Environment info
- `transformers` version: 4.17.0
- Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@NielsRogge

## Information

Model I am using: CANINE

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: Simple script that is passing some input to the model and gets logits.

The tasks I am working on is: NA, there is no task, I'm just trying to get logits from the model.

## To reproduce

Steps to reproduce the behavior:

1. Create CANINE tokenizer.
2. Create CANINE model for sequence classification with 2 labels.
3. Pass the same input to the model with multiple batch sizes.
4. For the same example, you will get different logits.

```python
import torch

from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('google/canine-c')
model = AutoModelForSequenceClassification.from_pretrained('google/canine-c', num_labels=2)

text = ['a', 'ab', 'abc', 'abcd', 'abcde', 'abcdef']

encoded = tokenizer(text, padding=True, return_tensors='pt')

model.eval()
with torch.no_grad():
    print(model(**encoded))

# Output logits:
# "a":      [0.1325, 0.1801]
# "ab":     [0.1764, 0.2585]
# "abc":    [0.0522, 0.0851]
# "abcd":   [0.0701, 0.1577]
# "abcde":  [0.1030, 0.0847]
# "abcdef": [0.0565, 0.1216]

encoded = tokenizer(text[:3], padding=True, return_tensors='pt')

model.eval()
with torch.no_grad():
    print(model(**encoded))

# Output logits:
# "a":   [0.1325, 0.1801]
# "ab":  [0.1764, 0.2585]
# "abc": [0.1694, 0.2958] -> This is different from the first ([0.0522, 0.0851]) output.

encoded = tokenizer(text[1:4], padding=True, return_tensors='pt')

model.eval()
with torch.no_grad():
    print(model(**encoded))

# Output logits:
# "ab":   [0.1764, 0.2585]
# "abc":  [0.1694, 0.2958] -> This is different from the first ([0.0522, 0.0851]) output.
# "abcd": [0.0552, 0.2409] -> This is different from the first ([0.0701, 0.1577]) output.
```

## Expected behavior

I expect the logits to be the same regardless the batch size passed to the model.

Note: I tested the same code with `microsoft/mdeberta-v3-base` model and I got the same output every time.<|||||>Hi @NielsRogge, could you please take a look ?.?<|||||>Hi,

Thanks for raising this issue. It probably has to do with the fact that CANINE internally downsamples the sequence length first (from characters to what the authors call "molecules"), before forwarding the input through the Transformer encoder. This isn't the case for regular Transformer encoders like BERT or mDeBERTa.

So when you provide: 

```
text = ['a', 'ab', 'abc', 'abcd', 'abcde', 'abcdef']
```

to `CanineTokenizer`, it will pad them all up to the length of 'abcdef'. As also the [CLS] and [PAD] tokens are added, each sequence will be padded up to 6 + 2 = 8 tokens (6 because there are 6 characters in 'abcdef').
However, when only taking:

```
text = ['a', 'ab', 'abc']
```
, then everything will be padded up to 3 + 2 = 5 tokens. 

The CANINE model has the `downsampling_rate` attribute in its config, which is set to 4. Hence, when forwarding the 8 tokens through the model, the 8 tokens will get downsampled to a sequence of 8/4 = 2 tokens, which are forward through the Transformer encoder. Hence the output of `CanineModel` will have a sequence length of 2.

However, when providing 5 tokens to the model, these will get downsampled to 5 // 2 = 2 tokens as well, but in a different way (due to the hashing technique, of which the details can be found in [this class](https://github.com/huggingface/transformers/blob/60d27b1f152c181705191765661967fef3016cef/src/transformers/models/canine/modeling_canine.py#L298)). This causes the result to be different.<|||||>I have two questions here:
1. What is the need of the attention mask if it will not discard the padding?
2. If I set the down sampling rate to 1, I will get the same output each time, right?

Thanks for your reply!<|||||>I just realized that if I want to set the down sampling rate to 1, I need to retrain the model, as the checkpoint will not be able to load. This is a problem actually (I mean the down sampling rate), because it affects the accuracy of the model.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.