Found the error on my own as I had to specify the column which had to be tokenized. The correct Tokenizer function would be:
def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True)

instead of
def tokenize(batch):
    return tokenizer(batch, padding=True, truncation=True)