You are most likely right and the inplace ReLU would manipulate the output of the incoming activation. You could copy-paste the VGG implementation and change these 1 and these lines of code 2 to nn.ReLU or alternatively replace them manually in the model.


I’m not sure I understand the question correctly. The gradients would be calculated for the parameters by default (not the activations). Which gradients would you like to calculate?