T5 Model : What is maximum sequence length that can be used with   pretrained T5 (3b model) checkpoint?
As the paper described, T5 uses a relative attention mechanism and the answer for this [issue](https://github.com/google-research/text-to-text-transfer-transformer/issues/273) says, T5 can use any sequence length were the only constraint is memory. 

According to this, can I use T5 to summarize inputs that have more than 512 tokens in a sequence?
Yes you can, but you should be aware that memory requirements quadruple when doubling the input sequence length for "normal" self-attention (as in T5).

So you will quickly run out of memory.

Here a snippet that shows that you can run input ids longer than `config.max_postion_embeddings`.

```python 
import torch
from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained("t5-base")
model.config.max_position_embeddings  # 512
input_ids = torch.tensor([600 * [0]])  # shape (1, 600)
model(input_ids, decoder_input_ids=input_ids)  # => no error
```

For more memory efficient models, you should take a look at `Reformer` and `Longformer`<|||||>I hope we will soon have these models ready for summarization<|||||>Thanks for the quick help. 

So basically, the T5 model in hugging face can handled arbitrary sequence length outputs right? 
So the second line (**model.config.max_position_embeddings**) basically shows the default max input seq length right ?

What do you think of the following code (Here I simply modify the tokenizer max_length):

```
model = T5ForConditionalGeneration.from_pretrained('t5-small')
 tokenizer = T5Tokenizer.from_pretrained('t5-small')
 t5_prepared_Text = "summarize: "+some_preprocess_text 
 tokenized_text = tokenizer.encode(t5_prepared_Text,  max_length=1024,return_tensors="pt")

 summary_ids = model.generate(tokenized_text,
                                    num_beams=4,
                                    no_repeat_ngram_size=2,
                                    min_length=30,
                                    max_length=100,
                                    early_stopping=True)


```
<|||||>Hi, I checked two summary outputs of T5, after using 1024 and 512  sequence lengths. I do not see any difference in generated summaries. Any idea for this behavior?<|||||>> Hi, I checked two summary outputs of T5, after using 1024 and 512 sequence lengths. I do not see any difference in generated summaries. Any idea for this behavior?

Hi I have the same question. Do you happen to figure out why?<|||||>Hi,

Those days I haven't had much of idea on huggiface models. Since we can add
any length as the input.. the main parameter should be minimum generation
length.

Try to change it.
<|||||>> Hi, Those days I haven't had much of idea on huggiface models. Since we can add any length as the input.. the main parameter should be minimum generation length. Try to change it.

I am still very new to huggiface. I have a pretty long text about 1500 words. The issue I was having is when I set max_length=512 or 1024, they kinda return the same summary. Do you know why?<|||||>I think it is because minimum length is unchanged. Regardless of the
input..  algorthm tries to generate a text until it gets the EOS (end of
sentence) token.  So it is common to get same length summary even if u add
few more sentence  to the original input.

On Mon, Feb 15, 2021, 16:40 mars997 <notifications@github.com> wrote:

> Hi, Those days I haven't had much of idea on huggiface models. Since we
> can add any length as the input.. the main parameter should be minimum
> generation length. Try to change it.
>
> I am still very new to huggiface. I have a pretty long text about 1500
> words. The issue I was having is when I set max_length=512 or 1024, they
> kinda return the same summary. Do you know why?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/5204#issuecomment-778917211>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGXCWKQKTGQML5LWTPLS7CJSLANCNFSM4OFG7QHA>
> .
>
<|||||>Hi, do we have to fine-tune the model when changing the ``model.config.max_position_embeddings``?<|||||>No really, cz T5 uses relative positional embeddings.<|||||>> I think it is because minimum length is unchanged. Regardless of the input.. algorthm tries to generate a text until it gets the EOS (end of sentence) token. So it is common to get same length summary even if u add few more sentence to the original input.
> […](#)
> On Mon, Feb 15, 2021, 16:40 mars997 ***@***.***> wrote: Hi, Those days I haven't had much of idea on huggiface models. Since we can add any length as the input.. the main parameter should be minimum generation length. Try to change it. I am still very new to huggiface. I have a pretty long text about 1500 words. The issue I was having is when I set max_length=512 or 1024, they kinda return the same summary. Do you know why? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <[#5204 (comment)](https://github.com/huggingface/transformers/issues/5204#issuecomment-778917211)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEA4FGXCWKQKTGQML5LWTPLS7CJSLANCNFSM4OFG7QHA> .

Personally, I think there is another reason: 

First, if you use the off-the-shelf T5-base model to summarize directly (i.e., no fine-tuning), a longer input would result in the same output as the original input. Because the T5-base model was pre-trained with `max_source_length==512`, those tokens exceeding `512 `may not be attended by the T5Attention layer. 

But after fine-tuning the T5-base model with a longer `max_source_length`, an input with a longer `max_source_length` perhaps gives you a different output than `512`.