I was able to see this through in the end.
There was an error in my ToTensor() transform. I guess that the final image not being normalized between 0 and 1 but between 0 and 255 was causing issue.
Thanks for the reply!