Thanks for your reply.
I got this traceback with torch.autograd.set_detect_anomaly(True).
[W python_anomaly_mode.cpp:104] Warning: Error detected in CudnnBatchNormBackward. Traceback of forward call that caused the error:
  File "<string>", line 1, in <module>
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/multiprocessing/spawn.py", line 118, in _main
    return self._bootstrap()
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/lr/wuhao/ssl-consistency-pytorch/remixmatch.py", line 220, in main_worker
    trainer(args, logger=logger)
  File "/home/lr/wuhao/ssl-consistency-pytorch/models/remixmatch/remixmatch.py", line 153, in train
    logits_rot = self.rot_classifier(x_ulb_s1_rot)
  File "/home/lr/wuhao/ssl-consistency-pytorch/models/nets/wrn.py", line 117, in rot_classify
    out = self.relu(self.bn1(out))
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 140, in forward
    self.weight, self.bias, bn_training, exponential_average_factor, self.eps)
  File "/home/lr/wuhao/anaconda3/envs/ssl/lib/python3.6/site-packages/torch/nn/functional.py", line 2147, in batch_norm
    input, weight, bias, running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled
 (function _print_stack)

It seems the relu part in the model cause this error. However, by checking the code of model, I found I had set relu with non-inplace operation as self.relu = nn.LeakyReLU(negative_slope=leaky_slope, inplace=False).
Why it still cause this error? Could you give me some hint?