You could create a custom Dataset as explained in this tutorial 11.
In the Dataset.__init__ method you could load the corresponding csv file and load each sample in __getitem__ lazily. To do so, you could index the csv file (e.g. via a pd.DataFrame), load (and transform) the corresponding image and create the target tensor.
Let me know, if you get stuck somewhere.