I have some conv nn and set np.random.seed manually, based on which I later fill in my starting weights of conv and fully-connected layers. To have everything deterministic. And it works. But then I added two MaxPool2d layers which I thought should be deterministic but turns out one of them is not. Basically these ar emy conv layers:
self.conv.add_module("conv_1", conv1)
self.conv.add_module("maxpool_1", torch.nn.MaxPool2d(2)) # 1
self.conv.add_module("relu_1", torch.nn.ReLU())

self.conv.add_module("conv_2", conv2)
self.conv.add_module("relu_2", torch.nn.ReLU())

self.conv.add_module("conv_3", conv3)
self.conv.add_module("maxpool_3", torch.nn.MaxPool2d(2)) # 2
self.conv.add_module("relu_3", torch.nn.ReLU())
I tested it and it turns out that if max pool layer #1 is present, the randomness is not deterministic. Why would that be?