Solved by ptrblck in post #2 
              
              
                Your loss is not decreasing either. 
Try to simplify your approach by trying to overfit a single sample. 
If the loss approaches zero and you succesfully overfitted your model, we could have a look at other possible errors. If that’s not possible, your model or training procedure has most likely a b…