Thank you, I modified the code to use the  in-place version of detach.
I match the detach generated with the input and store them in a list.
Next time, when the same input is presented, I return the stored detach as the output from this layer would be same as there was no back propagation.
But, with this approach my error rate remains quite high (98%) irrespective of the number of epochs or training size, where as, if I compute the output each time and then perform detach then the error rate goes down gradually.
My question is, if calling detach prevents back propagation and the parameters of the layer remain the same then for the same input presented again should produce the same output/detach and should match with my stored detach for that input. But, it seems it does not. Anything, wrong with this approach?
The modified code is
def forward(self, x):

    if not self.active:
        self.eval()
        out = self.conv1(F.relu(self.bn1(x)))
        out = self.conv2(F.relu(self.bn2(out)))

        out.detach_()

# store this detach in a list matching with the input. Next time, if there is a detach corresponding to the input then return that else compute the output/detach as done here and return it.

        return out