Have you tried to save in saved model format instead of h5?
Also just to check that your datagen pipeline is ok try to check if you can overfit the train+validation and use a more consistent batch _size.