I ran the previous command mentioned on the readme and it did output the binary file. However, another file is missing, token_counts.distilgpt2.pickle . How to generate that file as well?
On the token counter script, it returns IndexError: list assignment index out of range Someone know how to resolve that error?