Solved by ptrblck in post #2 
              
              
                I think usually these hyperparameters are determined heuristically, i.e. by testing a lot of values or finetuning “good known” values. There are some approaches to automate this, such as FasiAI’s learning rate finder. Also, the general approach would be “learning to learn” or meta learning and is a …