I created an NLP model on dataset a with torchtext. Now I tried to generate predictions for new cases but vocab.stoi is throwing a KeyError.
# Original TEXT field
# TEXT = Field(
#     sequential=True, 
#     tokenize=tokenizer, 
#     lower=True,
#     use_vocab = True,
#     #fix_length = 200,
#     pad_first = True
# ) 
TEXT = dill.load(open("field.dill", "rb"))

unl_datafields = [('Column1', None), ('Text', TEXT), ('Column2', None)]

unl = TabularDataset(
    path='./unl.csv', 
    format='csv',
    csv_reader_params={ 'delimiter': ';' },
    skip_header=True,
    fields=unl_datafields
)

unl_iter = Iterator(
    unl,
    batch_size=64,
    shuffle=False,
    sort=False,
    sort_within_batch=False
)

next(iter(unl_iter))

The code above throws the following error:
 334         if self.use_vocab:
    335             if self.sequential:
--> 336                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    337             else:
    338                 arr = [self.vocab.stoi[x] for x in arr]

KeyError: ' '

It seems like ’ ’ (a single space) cannot be found in vocab.stoi, which is suprising since the tokenizer splits on spaces. Moreover, I would like this to work with more words that are out of vocabulary. Can anyone give me some pointers on how to proceed?