How FNet handle PAD token?
How to avoid [FNet](https://huggingface.co/transformers/model_doc/fnet.html#transformers.FNetForPreTraining.forward) process `PAD` token as it has no `attention_mask` alternative argument in `forward` method?

```python
from transformers import FNetTokenizer, FNetModel
tokenizer = FNetTokenizer.from_pretrained('google/fnet-base')
model = FNetModel.from_pretrained('google/fnet-base')

features=tokenizer.encode(text="This is new encoder", max_length=16, padding="max_length",
                                           truncation=True, return_tensors="pt")
# tensor([[   4,  325,   65,  351, 1703,  242,   14,    5,    3,    3,    3,    3,
#            3,    3,    3,    3]])
```
Or does this model handle `PAD` token (with `token_id=3`) internally?
cc @gchhablani @patrickvonplaten <|||||>Good observation! The model actually processes the PAD token just like any other token, which means that padding an input of length 32 to 128 yields different results to just forwarded the input of length 32. For both pre-training and fine-tuning each sequence length was always padded to 512 tokens, so it's recommended to do the same for inference<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.