Thank you for your quick response and sorry for the confusion of using the verb.
But, I am not getting the expected result.
My question is, when we detach the output from the computation graph, the parameters of that layers are not updated in the back propagation. So, it should not matter if I compute the output again and call detach on that to prevent back propagation or return the stored detached output to be used by the next layer.
If I do not use the code
if (self.out[self.counter] is not None):
                detach = self.out[self.counter]
                self.counter+=1
                if(self.counter >= self.maxCounter):
                    self.counter = 0
					
				detach.volatile = False	
                return detach

and each time compute the output when self.active = False as given below, I get different expected result.
                out = self.conv1(F.relu(self.bn1(x)))
                out = self.conv2(F.relu(self.bn2(out)))
                out = torch.cat((x, out), 1)
# In place detach 			
                out.detach_()

                return out

My understanding was once I have detached output from back propagation, the outputs from both the code snippets above would be the same. But, it seems they are not.
My main purpose is to save on the output computation time of a layer, if I am not training that layer(removed  from back propagation).