The appropriate metric is exact-match, which is less lenient than ROUGE.

A TPU/TF trained T5-small scores 23% exact-match.
A GPU/HF trained T5-small barely gets to 15% ROUGE (which is more relaxed than exact-match).