It seems  to be the problem of pytorch. I found that some people encountered the same question when they implemented conv3d net.
RuntimeError: CuDNN error: CUDNN_STATUS_INTERNAL_ERROR when using 3D ver. of torchvision.models.resnet50  10
When set cudnn.benchmark=Flase, code can run normally.