Hello,

I don’t know if there are faster tokenizer, but would check what other repositories are using (e.g. HuggingFace).

Thank you for your reply. I appreciate your efforts 

This would iterate the Dataset and DataLoader for an entire epoch and could be used to isolate the indexing issue.

It still running very slow which is not very practical so I am going to change my strategy!
Thanks anyway!