Turns out the issue was with h5py and DataLoader. Using 2 or more threads with DataLoader when reading from an hdf5 input results in corrupted data. The read data was randomly corrupted and that resulted in errors with the network, with losses going to nan, or performing illegal indexing.