Further reduce the number of alls to head for cached objects
# What does this PR do?

This PR completes #18534 and leverages the cache system of files that do not exist at a given commit in a repo introduced in the last release of `huggingface_hub` (by this [PR](https://github.com/huggingface/huggingface_hub/pull/986)) to further reduce the numbers of calls to the API when trying to load configurations/models/tokenizers/pipelines to just 1 call **every time** the object is cached and the current commit is the same one as the distant repo for the given revision.

cc @Narsil 



_The documentation is not available anymore as the PR was closed or merged._<|||||>and also cc @Wauplin :)<|||||>Yes, my plan was to port this to `hugginface_hub` next, along with the commi_hash argument (which does not exist there yet), to then be able to use the function of `huggingface_hub` after the next release!

Thanks for the reviews, will address comments later this morning.