I would assume that simple slicing is faster than calling into the __getitem__ method of your Dataset and stacking the samples to a tensor.
The slowdown of 20x is quite large and I wouldn’t expect it. However, did you try different number of workers for your DataLoader approach and compared it to the manual slicing approach?
The main advantage of using the DataLoader is e.g., that you can use multiple processes to prefetch the next batch, while your main training loop is busy with the model training.
Also, you can easily shuffle, provide a custom sampler, a custom collate_fn, use pinned memory, and can easily implement lazy loading of the samples in your Dataset.
If you can store the data in memory and don’t need the advantages of the Dataset and DataLoader, your approach seems completely fine.