Hello!
Is there a recommended/easy way to find the total number of tokens in a huggingface dataset?
I know it’s possible to use FreqDist in nltk, but it would be great if there was an implementation in datasets.
Thanks in advance.