Oh all right! In that case creating and training different models for each input size is the expected approach, and you just want to create models more automatically than defining each layer’s size by hand… I understand now 
Your approach should work, it all depends on how your model is defined. If you don’t use padding as a way to keep the input sizes of layers, like ResNet does (the size of the features in the end of the network are much smaller than the input images), I think you will probably need to figure out a formula for handling different heights depending on your model’s layers.
The dataset could handle inputs dynamically also, depends on the code…