Based on the code snippet it looks indeed a bit fishy.
Could you post an executable code snippet, so that we could reproduce this issue and debug it, please?
Also, which Pytorch and CUDA versions are you using?