Not sure, the leaky_relu doesnt seem to have that problem. The upsample is just local replication, so I’m not sure that’s the problem.
The issue is that it drops to 10%. The way I fixed it is to have:
    def forward(self, x):
        x = self.prelu_ac[0](self.bn[0](F.max_pool2d(self.conv1(x), 2)))
        #x = self.bn[0](x)
        x = F.upsample_bilinear(x, size=(16, 16))
        x = self.prelu_ac[1](self.bn[1](F.max_pool2d(self.conv2_drop(self.conv2(x)), 2)))
        #x = self.bn[1](x)
        x = F.upsample_bilinear(x, size=(16, 16))
        x = x.view(-1, 5120)
        x = self.prelu_ac[2](self.bn[2](self.drop1(self.fc1(x))))
        #x = self.bn[2](x)
        x = F.dropout(x, training=self.training)
        x = self.prelu_ac[3](self.bn[3](self.drop2(self.fcmid(x))))
        #x = self.bn[3](x)
        x = F.dropout(x, training=self.training)
        x = self.prelu_ac[4](self.bn[4](self.fc2(x)))
        #x = self.bn[4](x)
        return F.log_softmax(x)

Any tips for improvement? Should I drop the upsample? I think it helps the second conv layer