It might not work since t5-large and t5-small have different hiddden_size, attention_head, feed-forward dim, the layers are  sized differently, so it’s not possible to copy weights when the sizes don’t match.
cc. @sshleifer