Thank you for your reply!
I ran it with CUDA_LAUNCH_BLOCKING=1 and got the following error:
THCudaCheck FAIL file=/tmp/pip-req-build-jh50bw28/aten/src/THCUNN/vol2col.h line=64 error=77 : an illegal 
memory access was encountered
Traceback (most recent call last):
  File "train_segmentation.py", line 213, in <module>
    train(args)
  File "train_segmentation.py", line 123, in train
    model.forward(split='train')
  File "/myproject/models/feedforward_seg_model.py", line 127, in forward
    self.prediction = self.net(self.input)
  File "/myhome/.conda/envs/pytorch_seg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/myproject/models/networks/unet_CT_V_edge_3D.py", line 67, in forward
    x = self.aspp(up1, acts)
  File "/myhome/.conda/envs/pytorch_seg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/myproject/models/networks/unet_CT_V_edge_3D.py", line 327, in forward
    y = f(x)
  File "/myhome/.conda/envs/pytorch_seg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/myhome/.conda/envs/pytorch_seg/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/myhome/.conda/envs/pytorch_seg/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/myhome/.conda/envs/pytorch_seg/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 476, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /tmp/pip-req-build-jh50bw28/aten/src/THCUNN/vol2col.h:64

But, weirdly, when I wrote a simple network with only 1 ASPP model above, and feed in randomly generated samples np.random.randn(32, 144, 144, 144).astype(np.float32), it worked.
I am very confused. Could you help me with the above error message?