You PyTorch code looks generally alright, although you could lazily load the data using a custom Dataset as described in this tutorial 18.
Inside this dataset you can also assign your custom labels in the __getitem__ method. Let me know, if you get stuck.