That example helps. I will try looking in that direction.
A related question, if I iterate over the data multiple times (like when I have multiple epochs), will the myTokenizer and the processing functions run every time I iterate over the whole data, or only at the first iteration? Because, if all the tokenizing and processing is performed every time, then wouldn’t it be better to first store all of the processed data in a matrix and then perform multiple iterations using it?