I assume that the output of the Dataset.__getitem__ would still yield the same objects regardless how the data is loaded internally (from the JSON file, H5, or directly as images).
If that’s the case you could write different Dataset implementations and use ConcatDatset to create a single one before passing them to the DataLoader.