What GPUs are you using?
Is only the first CUDA call taking a long time?
In the past this was due to a mismatch of your CUDA version which resulted in PyTorch being recompiled for your GPU. Could you try it with CUDA8 and see, if it still takes that long?