Solved by Lost_Wanderer in post #4 
              
              
                This was solved using a lower learning rate. The initial learning rate was 0.05, while I changed it to 0.005. 
My question now is: Why did lowering the learning rate solve the problem in this case? In the other issue I had (the issue you linked), I had a deeper model (ResNet 50; in the book they use…