Hi all,
I’m currently trying to load in fastai’s version of the IMDB dataset, to learn how to build a Dataset from a folder of .txt's. I’m preparing my data with the following:
# Downloading the dataset
from fastai.data.external import untar_data, URLs
from fastai.data.transforms import get_files

path = untar_data(URLs.IMDB, dest='./IMDB')

From there I can get all the training .txt's with:
texts = get_files(path/'train', extensions='.txt')
texts = [str(t) for t in texts]

In turn, this is a list of 25,000 text files. However, when I use the load_dataset api to bring this in, suddenly my dataset has 25,682 items! Can anyone help me figure out why? This is an issue as I need to use add_column to add in a label, and there is a mismatch between the number of actual training items vs the ones Datasets picked up. Here is how I’m building the dataset:
dset = load_dataset('text', data_files={'train':texts})

TIA!