How to implement generate function for seperate encoder decoder T5 model?
I was working on optimizing the T5 model. The version of the transformer I am using is  4.8. For optimization, I separated the model into encoder and decoder with LM(Language modeling) head. Earlier for generation, I just passed input_ids, attention_mask,max_length, and num_beams. But now since the model is separated into encoder and decoder I can't use generate function. But I am able to generate logits using the below code:

encoder_last_hidden_state = t5_trt_encoder(input_ids=input_ids)
outputs = t5_trt_decoder(input_ids, encoder_last_hidden_state)

But how can I use these logits to generate sequences?. I mean I am unclear how can I use the encoder and decoder to write a class which supports generate function?

----

note: edited by @stas00 to remove a dozen of tags 
Hi! Thanks for the issue. But please don't tag every maintainer on an issue. In the issue template you can find who to tag depending on the model or feature. 
Also for such general questions please use the [forum ](https://discuss.huggingface.co/). We use issues for bug reports and feature requests. Thank you !
<|||||>Hi @patil-suraj. Thanks for letting me know about the community. I was trying to optimize the T5 based QA-QG model from your repository using https://github.com/NVIDIA/TensorRT/tree/main/demo/HuggingFace repo. Here they have separated the model into encoder and decoder with LM(Language model) for optimization. But when I am doing inference in the following way:
max_length = 64

decoder_input_ids = torch.full(
    (1, 1), tokenizer.convert_tokens_to_ids(tokenizer.pad_token), dtype=torch.int32
)
encoder_last_hidden_state = t5_trt_encoder(input_ids=input_ids)

outputs = t5_trt_decoder.greedy_search(
            input_ids=decoder_input_ids,
            encoder_hidden_states=encoder_last_hidden_state,
            stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length)])
        )
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

I am getting an empty result. I tried with an unoptimized torch model too but the results are the same. Hence I thought maybe that can be resolved if somehow I am able to use logits from these in generate function. Hence I asked the question.
The same prediction method works for translation case.

The tokenizer is the same as Valhalla/t5-small-qa-qg-hl

Please help.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.