The problem was in mismatch of CUDA and Pytorch versions. I had Pytorch -v: 1.3.1+cu92.
However, when I tried run training with installed CUDA 9.2 I’ve had the similar problem.
It seems like Turing cards is only supported by CUDA 10.X:

CUDA 10 is the first version of CUDA to support the new NVIDIA Turing architecture
https://devblogs.nvidia.com/cuda-10-features-revealed/ 8

With the latest version of CUDA and Pytorch everything works correctly and most importantly fast!
Thanks