zhuser:

I do agree that this has to be the case and I would like to attract your attention to the point that printing out the auto_grad status of the different layers of the model using “.named_parameters()” and then applying “.requires_grad” shows “requires_grad=True”. Given that the whole block is inside “with torch.no_grad():” I expected to get “requires_grad=False”! Is this an expected behaviour? Could it be that the gradient operation is still in action?


That’s expected, as the with torch.no_grad() block won’t switch any attributes of the model parameters, but will instead make sure that new operations performed inside this block won’t be tracked. Also, if you don’t call backward() on the output or any loss tensor and optimizer.step(), the parameters won’t be changed through the forward pass only.
Thanks for the code.
In Encoding you are creating new, randomly initialized, Residual modules inside the forward method, which will cause the difference in the outputs.