Longformer output_hidden_states=True outputs sequence length=512 for all inputs of different lengths
Can someone explain why cc[2][0].shape != dd[2][0].shape?

>>> from transformers import LongformerModel, RobertaModel
>>> import torch
>>> from transformers import RobertaTokenizer
>>> tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
>>> aa = RobertaModel.from_pretrained('roberta-base', return_dict=True)
>>> bb = LongformerModel.from_pretrained('allenai/longformer-large-4096', return_dict=True)
Some weights of LongformerModel were not initialized from the model checkpoint at allenai/longformer-large-4096 and are newly initialized: ['longformer.embeddings.position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> cc = aa(**inputs, output_hidden_states=True)
>>> cc[2][0].shape
torch.Size([1, 8, 768])
>>> dd = bb(**inputs, output_hidden_states=True)
>>> dd[2][0].shape
torch.Size([1, 512, 1024])
>>> dd[0].shape
torch.Size([1, 8, 1024])

When I use longformer-base-4096, it works well.<|||||>The reason is that `LongformerModel` pads its input to the `window_size` configuration param, which is 512 for `allenai/longformer-large-4096`. Therefore all `hidden_states` are of length 512. We could think about cutting the hidden states to the actual inputt length...<|||||>Gotcha. But I also tested with longformer-base. It works fine with hidden states cut. Thank you.