Variable batch size wouldn’t be bad per se that said the data unfortunately is nearly delta shape distributed 
As you said I think I will need to add another stage which converts the data from HDF5 to patches and then feed that data into pytorch.
Do you have a recommendation on what dataset interface to use? Did you experience any problems using the ImageFolder dataset with lots of image patches (IO wise)?
Thanks in advance!