This is a problem that I have gotten: full info on this thread 4 I had a quick question -
I have constructed Dataset object using numpy arrays, however when I use this:-
def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=2000)
  return encodings

train_encoded_dataset = train_dataset.map(tok, batched=True)
val_encoded_dataset = val_dataset.map(tok, batched=True)

and I explore my train_encoded_dataset, I see that when trying to view the source sequence:
>>> train_encoded_dataset
>>>Dataset({
    features: ['attention_mask', 'input_ids', 'src', 'tgt'],
    num_rows: 4572
})
>>> train_encoded_dataset['src'][0]

The output of this last command produces text that is completely raw (basically untokenized) string- (like: ‘lorem ipsum…’) which is expected since I didn’t call tokenizer.tokenize
So does anyone have any idea how to get the model to tokenized as well? I tried a few obvious ways, but it didn’t yield anything