Update :
I checked with separate dataloaders for the separate dataset classes whom I call inside my fused dataset class.
There is indeed a mismatch, in first dataloader a 5D numpy array is converted to torch.FloatTensor while other it is torch.DoubleTensor