This might be expected, since the DataLoader will add overhead to the already loaded tensors by e.g.:

shuffling the indices using a sampler
indexing the TensorDataset with each index separately
creating the batch by calling into the collate_fn


This workflow is useful, if you want to lazily load the dataset and process each sample via data augmentation techniques.
Since your dataset is already loaded, indexing would be the faster approach.