fastT5 now works with transformers>=4.12.5 (this is the latest version I have tested). It also has some tweaks to the quantization settings that should speed it up a bit more.
I can confirm it works with byt5 now, and it should work with mt5 as well (except for needing to change the ConditionalGeneration class as noted above). However,  byt5 is substantially slower due to both the larger encoder and needing one decoding step per character
EDIT: I just opened a PR to explicitly support mt5 as well. I did some limited testing and it seems to work.