I could not replicate it on google colab. I was able to get the speeds of both models to run the same. I think it was some type of caching issue. I have no real idea where to start to actually debug it.