I think the approach of using your already valid .csv file and keeping the folder structure sounds reasonable and I would avoid moving around the data.
To use the csv data, I would recommend to load it in your Dataset.__init__ method with e.g. pandas via pd.read_csv and use the index in Dataset.__getitem__(self, index) to get the current file path and label.
I’m not sure how your csv is structured, but you can index the rows and columns of the pd.DataFrame easily.
Once you have the image path, you could load it with PIL and transform it to a tensor.
The label will most likely be returned as a numpy array/scalar, so you could use torch.from_numpy() to create the target tensor.