How to get T5 encoder&decoder's hidden states and keep requires_grad=True
transformers version:
4.15.0

Hi! I am making a custom T5 generation model, and I have some problems. In my model, I want to take out the decoder's last hidden states and encoder's last hidden states to do some other operations, and these operations will be used to fine-tune the encoder and decoder. So I need last_hidden_state's requires_grad=True and is_leaf=False.
I get the encoder_last_hidden_states and decoder_last_hidden_states in the following code:

        outputs1 = self.T5model1(
        input_ids=batch["source_ids"],
        attention_mask=batch["source_mask"],
        labels=lm_labels,
        decoder_attention_mask=batch['target_mask'],
        output_hidden_states=True,
        )
        decoder_last_hidden_states=outputs1['decoder_hidden_states'][-1]
        encoder_last_hidden_states=outputs1['encoder_last_hidden_state']
But I found that the requires_grad of decoder_last_hidden_states and encoder_last_hidden_states are False and is_leaf of them is True. It looks like the gradient doesn't propagate. 
How do I get these vectors so that I can finetune the full six-layer encoder and six-layer decoder.
