if __name__ == '__main__':
    torch.backends.cudnn.deterministic = True
    model = resnet50(num_classes=128)
    model.cuda()
    model.eval()

    x = torch.randn(20,3,224,224).cuda()
    x1 = x.reshape([10,2,3,224,224])[:,0,:,:,:]
    x2 = x.reshape([10,2,3,224,224])[:,1,:,:,:]

    out1 = model(x1)
    out2 = model(x2)
    out = model(x)
    

    out = out.reshape(10, 2, -1).transpose(1,2)
    out_ = torch.stack([out1, out2], dim=2)
    err = (out - out_).abs().max()
    print(err)

In above code, out and out_ shall be same logically. (Note that I use the official resnet implementation in torchvision/models/resnet.py)
However, I encountered some strange phenomenon:

v100-gpu + ‘‘1.3.1+cu92’’-pytorch-version, err = 0 all the time (expected)
t4-gpu + ‘1.4.0’-pytorch-version, err is in the order of 1e-5 (unexpected)
After I comment either fc or avgpool (or comment both) in forward func of resnet like below:

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        # x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        # x = self.fc(x)

The err changes to 0 in t4-cuda10 environment.
Is this just caused by the difference of the cuda version? Is there any way to control the nodeterministic result with cuda10?