Hello,
I guess this is more a python for loop issue and/or a Colab one but as I tested it with a HF tokenizer, I’m sending this question to this forum: Why the HF tokenizer time is bigger when launched just once?
I published a colab notebook 2 to explain this issue that is showed in the following graph:

If I launch just one time the tokenizer on a text, it will always takes a much bigger time than the average time of x tokenizations of the same text. Strange, no?
Configuration

transformers version: 4.11.3
tokenizer from the model bert-base-uncased

importation code of the tokenizer:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)