Could you share the code you are using to load the data?
You collate function would be of interest.
I assume the first dimension corresponds to the number of samples in the current file?
Is it not possible to load the samples in a Dataset without a custom collate_fn?