I don’t know if there are faster tokenizer, but would check what other repositories are using (e.g. HuggingFace).



 MABtt:

It would be VERY helpful if you could tell me more about your proposed solution, I didn’t get what do you mean by


Something like this:
for data, target in dataset:
    print(data.shape)

for data, target in loader:
    print(data.shape)

This would iterate the Dataset and DataLoader for an entire epoch and could be used to isolate the indexing issue.