Hi @KURI, have you tried profiling both models to see which layers are the bottlenecks?  That would help narrow down the issue.  PyTorch Profiler — PyTorch Tutorials 1.7.1 documentation 8 has an example.