Solved by ptrblck in post #2 
              
              
                I don’t think there is a simple answer to this question. 

I think that’s unfortunately still often the case and you will read in a lot of papers that the training hyperparameters were “tuned” or “heuristically determined”. 
You could have a look at e.g. FastAI which explains the best practices for …