sgugger:

I think it’s because the “text” loader creates a new sample for each “\n” it sees, so the texts you have that contain some of those are then split into several samples. @lhoestq or @albertvillanova if you could conifrm?


I don’t think so, when I tried checking for that I still got 25,000. Or a better way to put that is this returns zero:
from fastcore.xtras import open_file

count = 0
for text in texts:
    t = open_file(text).read()
    if '\n' in t or '\r' in t: count += 1




 sgugger:

PS: it would be easier to just do dsets = load_dataset('imdb') :-p


Of course it would be! However I’m trying to write a high-level data API for adaptnlp currently, so I’m only using IMDB as a situational test case 
Edit: Trying a new way to verify, will update with those results
Aha! @sgugger thank you! There were some hidden \x85 characters, which is the source of the breakage.
I can work with that now. Thank you! 
(If you have recommendations for fixes, I’m all ears, I was just going to take that into account while mapping labels from folder names)