@SimonW If I do out += residual.detach(), it will block the gradient during training process. What I want is, block the gradient during test time only, which looks like:
ResNet.eval()
pred = ResNet(input_variable)
ce = nn.CrossEntropyLoss()
loss = ce(pred, GT_label)
loss.backward()

In the loss.backward(), I would like to block the gradient from residual. For self.downsample is not None, I can register a backward hook founction to change the gradient. However, if self.downsample is None, I cannot do that. Then, how I can still block the gradient from residual