Solved by ptrblck in post #2 
              
              
                I guess you are observing the adaptive learning rates for different parameters of optim.Adam. 
Try to use a simpler optimizer like optim.SGD and run it again.