Ok, I see.
Because most layers were frozen and just a couple being trained, I expected the pre-trained model to be faster than my “from scratch model”.
Thanks for your answer. I will try LR scheduler.