Solved by Wertiz in post #3 
              
              
                I was able to see this through in the end. 
There was an error in my ToTensor() transform. I guess that the final image not being normalized between 0 and 1 but between 0 and 255 was causing issue. 
Thanks for the reply!