I trained a BPE tokenizer using the wiki-text and now I’m trying to use this tokenizer on a custom dataset from a csv file. What I want to achieve is additional of feature columns in my dataset. But the dataset.map is giving error.

Screenshot 2021-06-23 at 5.47.51 PM730×904 152 KB