Hi @ptrblck
I think I have figured the mishap I was making.
I did not use Batchnorm after the lastConv layers and without normalization, I guess that is the reason the outputs were rocketing.
        x = self.bnl2(self.lastConv2(x))
        x = self.relu(x)
        x = self.bnl1(self.lastConv1(x))
        x = self.relu(x)
        x = self.bnl0(self.lastConv0(x))
        x = self.relu(x)
        x = x.view(x.size(0), -1)
        x = self.dropout(x)
        x = self.fc1(x)
        x = self.dropout(x) %40
        x = self.fc2(x)
        return x


Though I guess the network overfits right after epoch: 2 but this is kinda normal problem.
Let me know your thoughts… I will provide more details if you need.
Thanks