hey @Neel-Gupta, could you share a minimal example of the Dataset object you’re working with?



 Neel-Gupta:

The output of this last command produces text that is completely raw (basically untokenized) string- (like: ‘lorem ipsum…’) which is expected since I didn’t call tokenizer.tokenize
So does anyone have any idea how to get the model to tokenized as well? I tried a few obvious ways, but it didn’t yield anything


just so i understand, you’re saying that the tokenizer is not tokenizing the strings in the src field right?