Hi,
I am a bit confused about where to exactly apply dropout in CNN network.
In the below model I applied dropout in both of the Conv layers and also in the linear layer.
But I am not sure whether I need to apply it. After ReLu? or before ReLu ? in linear layers.
And also I am not sure if I implemented dropout in correct place in Conv layers.
I am experimenting on dropout mc outputs of the CNN model : uncertainty metrics
I got different mean confidence values and uncertainty values, when I used dropout before or after the F.relu for fc1.
class CNN_dropout(nn.Module):
    def __init__(self):
        super(CNN_dropout, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)
        self.drop_layer = nn.Dropout(p=0.5)

    def last_hidden_layer_output(self, x):
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv2(x))), 2)
        x = x.view(-1, 320)
        x = F.relu(self.drop_layer(self.fc1(x)))
        return x

    def forward(self, x):
        x = self.last_hidden_layer_output(x)
        x = self.fc2(x)
        return x

My experiment results differ when I switch
from
    def last_hidden_layer_output(self, x):
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv2(x))), 2)
        x = x.view(-1, 320)
        x = F.relu(self.drop_layer(self.fc1(x)))
        return x

to
    def last_hidden_layer_output(self, x):
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)
        x = F.max_pool2d(F.relu(self.drop_layer(self.conv2(x))), 2)
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.drop_layer(x)
        return x

If I am sure where exactly to apply dropout in linear layer, maybe I also need to change the place of dropout ın conversation layers too?
To me, these seem to be a better choice but I am not sure unfortunately.
x = F.relu(self.drop_layer(self.fc1(x)))

x = F.max_pool2d(F.relu(self.drop_layer(self.conv1(x))), 2)