Make more test models smaller
# What does this PR do?

This PR continues on the work of #24824 by fixing the models used in more common tests. This is big enough to be reviewed, I'll do the last ones in a separate PR :-)
_The documentation is not available anymore as the PR was closed or merged._<|||||>Yes we could go even smaller if we wanted to, but it's hard for big encoer/decoder models with backbones.