Thanks a lot for the quick reply! Yeah, the tokenizer is not tokenizing those strings, maybe because I didn’t call the tokenize method?
Anyways, plugging these variables do the trick for reproduction
train_text = np.array(['a foxy', 'b ball', 'c cats r bad'])
train_label = np.array([1,2,3])
val_text = np.array(['a foxy', 'r c cats'])
val_label = np.array([1, 2])

if you can’t reproduce the issue with the pre-trained longformer (base) tokenizer, then I can provide you my model - but I doubt it won’t since I have used that same one 
tokenizer = AutoTokenizer.from_pretrained(".....", use_fast=True, truncation=True, padding=True, max_length=2000)