as I understand it, those are two very different goals.


splitting up a dataset into test/train/validation/etc
as alluded to in the earlier post you can do something with the example from vision


tokenization
this is a step earlier in the process, in which you are creating your dataset of tokens.  Then after you have your dataset of tokens, split it up randomly…


OR
if you want to perform the tokenization as a means for splitting… just replace my example in which I split the array… and use your tokenization strategy instead.
Hope I am understanding you,