Update, I believe I may be able to make this work if using torchvision.datasets.DatasetFolder and setting my own loader for FITS extensions.
However when trying to use this class I get the following error:
AttributeError: module 'torchvision.datasets' has no attribute 'DatasetFolder'
Is DatasetFolder actually supported by torchvision at this point in time?