I was trying an ASPP-like module with dilated conv3d:
class _AtrousSpatialPyramidPoolingModule(nn.Module):

    def __init__(self, in_dim, reduction_dim=256, output_stride=16, rates=[6, 12, 18]):
        super(_AtrousSpatialPyramidPoolingModule, self).__init__()

        self.features = []
        self.features.append(
            nn.Sequential(nn.Conv3d(in_dim, reduction_dim, kernel_size=1, bias=False),
                          BatchNorm3d(reduction_dim),
                          nn.ReLU(inplace=True)))
        
        for r in rates:
            self.features.append(nn.Sequential(
                nn.Conv3d(in_dim, reduction_dim, kernel_size=3,
                          dilation=r, padding=r, bias=False),
                BatchNorm3d(reduction_dim),
                nn.ReLU(inplace=True)
            ))
        self.features = torch.nn.ModuleList(self.features)

    def forward(self, x):
        for f in self.features:
            y = f(x)
            out = torch.cat((out, y), 1)
        return out

It worked on CPU, but I got the following error on GPU:
RuntimeError: cublas runtime error : library not initialized at /tmp/pip-req-build-ocx5vxk7/aten/src/THC/THCGeneral.cpp:216

at the line y = f(x) when it ran to the second loop (dilated conv3d).
But when I changed dilated convs to common convs, it also worked:
        for r in rates:
            self.features.append(nn.Sequential(
                nn.Conv3d(in_dim, reduction_dim, kernel_size=3,
                          dilation=1, padding=1, bias=False),
                BatchNorm3d(reduction_dim),
                nn.ReLU(inplace=True)
            ))
        self.features = torch.nn.ModuleList(self.features)

My environment:

4 Tesla V100-SXM2-32GB GPUs, no OOM scenario monitored (as I see it)
cudatoolkit 10.0.130
pytorch 1.1.0 (conda, also tried Pytorch 1.3.1)
python 3.6.7

Thank you for your time!