ptrblck:

Is the slowdown only happening on scripted models or any GPU workload?


Apologies, I didn’t catch that the point of your question initially. It is only on scripted models.
Just to check if I understand correctly now:
I have the training function of my model wrapped in some boilerplate code (partly pytorch-lightning, but it should be irrelevant). I have the option to trace this function and then use it for training, or to skip this step and train with all python host code.
My original question was based on what i observed at inference time (forward function has been exported and re-imported) but indeed even in training i observe a delay on the second iteration only when the forward function is traced, whereas with all python host code the times are even.



 ptrblck:




 michele-arrival:

Btw is there any way to check that pytorch is loading the correct cuda libraries?


You could check the used CUDA version via torch.version.cuda and depending on the way you’ve installed it (binaries or source build) it can be either statically or dynamically linked.


cuda = '10.2' in torch/version.py
It’s installed from anaconda, so it should be all statically linked i believe?
Thanks!