I have replicated the test with a more realistic task, encoding the sequences with BERT from pytorch-transformers on my CPU without any threading except num_workers in the dataloader.
Outcome is the same, using batching via the dataset is at least 10x faster than using a dataloader.
I am 100% sure  I am doing something wrong, but my dataset class is quite standard, except that I have to open and close a database file, and tokenize+pad in the getitem function, but of course that is the same whether I use dataloader or custom batches.
Thoughts?