Finishing tidying keys to ignore on load
# What does this PR do?

A couple of tests are failing on main due to #24505 (and PRs merged between the start of the work on this branch and its merge). This should fix all of them.

Note: will merge as soon as CI is green to make main green again.
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24535). All of your documentation changes will be reflected on that endpoint.