I have not used those two specific models before, however without a small reproducible example of your code is impossible to provide personalised feedback. You didn’t mention what you’re doing or using the model for, but I’ll assume you’re doing inference (not training or fine tuning) based on the run times you reported.
Ultimately doing inference on a CPU will be much slower than on a GPU, there is absolutely no way around it (although the time difference if you’re doing training in a   CPU vs GPU is even bigger). Whether there is still some margin for CPU optimisation in your specific case, is something you would need to share the code for the community to be able to comment on.
The only thing that comes to my mind right now (without knowing anything at all about your code), is to only initialise the model once, and then do inference on the same instance, so avoid calling something like
trained_model = AutoModel.from_pretrained(...)

at every iteration inside a for loop, as that would be very detrimental, but without seeing the code I can’t say much else.