Hi ptrblck,
Pytorch is installed from the source.
During the first call cublas handle creation is taking time, the same handle is used in the second call. I feel this is the reason for higher latency in the first call.