Might be related to the fast tokenizer, which is a multiprocessing-enabled Rust tokenizer. I’d suggest to have on tokenizer in a separate process and use a queue to request tokenization. You can then run different models in different processes.
However:

First thing to speed everything up is using a single model and large batches, rather than doing line-per-line operations. That is incredibly slow.
If you have a GPU available, use it. It’ll be faster.