Hey @pierreguillou,
could you share how you created your Training Job (python estiomator + hyperparameters) and the training script you use?
Iin addition to this, I am not sure if T5 is well supported yet. As mentioned in my blog

The Amazon Training Compiler works best with Encoder Type models, like BERT , RoBERTa , ALBERT , DistilBERT .

The Training Compiler currently works best with encoder-type models. In the test I ran a couple of weeks back T5 performed worse than the default training.