Solved by nielsr in post #4 
              
              
                Yes feature extractors also have a from_pretrained method, to just load the same configuration as the one of a particular checkpoint on the hub. 
e.g. if you do ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224"), it will make sure the size attribute of the feature extractor is set to…