Oh I know why. So for dataloader, since all your dataset give is the __getitem__, what it does is to retrieve a bunch of tensors at different indices, and then cat them. It is done this way so it can be very general and work on any dataset.
However, in case of TensorDataset, you have all data in memory, and can do much more efficiently. You may

Shuffle the entire tensor before hand, and then do contiguous slicing
Or slower, but still faster than the DataLoader way, use index_select (or advanced indexing).

My guess is that if your code mimic the DataLoader behavior, they will be of similar speed.