Just confirming that I’m having the exact same issue on RTX 2070 (CUDA 10.0, Pytorch 1.0.0) with my first BatchNorm2d layer in my GAN generator. Looks roughly like this:
# bs=64, latent_sz=128, init_sz=8
z = torch.randn(bs, latent_sz).cuda()
generator(z) # generator = Generator().cuda()

# In generator forward(self, x):
x = self.fc(x).view(bs, 128, 8, 8) # fc is nn.Linear(latent_sz, hidden_sz * init_sz**2)
x = self.bn1(x)      # see nn.Sequential() below for init. Real code puts x through the Sequential
...

If I remove this BatchNorm2d (self.bn1 in the example) from the nn.Sequential it works. The nn.Sequential looks as follows:
# hidden_sz=128, init_sz=8
nn.Sequential(
            nn.BatchNorm2d(hidden_sz),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(hidden_sz, hidden_sz, 3, stride=1, padding=1),
            nn.BatchNorm2d(hidden_sz, 0.8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(hidden_sz, hidden_sz / 2, 3, stride=1, padding=1),
            nn.BatchNorm2d(hidden_sz / 2, 0.8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(hidden_sz / 2, nc, 3, stride=1, padding=1),
            nn.Tanh()
        )
# All divisions hidden_sz / x  is actually hidden_sz // x, 
# I changed it here so it is not displayed as comment in the forum