Note, this does not happen when the batch size is small (actually this occurred when I was using batch size of 2 in segmentation, so I think it is more to do with how much memory is being used rather than just batch size).
On 10.2
>>> import torch
>>> import torch.nn as nn
>>> torch.__version__
'1.7.1'
>>> torch.version.cuda
'10.2'
>>> torch.manual_seed(42)
<torch._C.Generator object at 0x7f32829412b0>
>>> l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
>>> model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
>>> torch.manual_seed(42)
<torch._C.Generator object at 0x7f32829412b0>
>>> a = torch.randn(4, 3, 224, 224)
>>> b = model(a.to('cuda:0')).squeeze()
>>> b
tensor([0.0109, 0.0109, 0.0109, 0.0109], device='cuda:0',
       grad_fn=<SqueezeBackward0>)

On 11.0
>>> import torch
>>> import torch.nn as nn
>>> torch.__version__
'1.7.1'
>>> torch.version.cuda
'11.0'
>>> torch.manual_seed(42)
<torch._C.Generator object at 0x7f1da0aaa290>
>>> l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
>>> model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
>>> torch.manual_seed(42)
<torch._C.Generator object at 0x7f1da0aaa290>
>>> a = torch.randn(4, 3, 224, 224)
>>> b = model(a.to('cuda:0')).squeeze()
>>> b
tensor([0.0109, 0.0109, 0.0109, 0.0109], device='cuda:0',
       grad_fn=<SqueezeBackward0>)