You can load .mat files with scipy.io.loadmat 20.
If you could create one .csv file containing the image path and both labels, we could write a Dataset and load all corresponding images and labels using this file without moving your data.