Are only used parameters updated?
When A is a module with multiple sub-modules, where only a sub-set of the sub-modules are used dependent on the input, for example, A is defined as follows, <SCODE>class A(nn.Module):
    def __init__(self):
        self.common = nn.Linear(100, 50)
        self.module1 = nn.Linear(50,30)
        self.module2 = nn.Linear(50,2)
    def forward(self, input, idx):
        commonOut = self.common(input)
        if idx == 0:
            return self.module1(commonOut)
        else:
            return self.module2(commonOut)

opt = torch.optim.Adam(A.parameters()) 
<ECODE> If it does, is there any way I can get the list of parameters that are actually updated? Thanks
supakjk
No, we don’t do that. There’s no way how the optimizer would know which parameters were or weren’t used, and there’s no possible way to find out about that. It would require us to impose some strict requirements, that don’t really make sense in most use cases, for only minor improvements in other. Why is it such a problem that these parameters are getting updated? Is your script slow? It shouldn’t add a lot of overhead.
apaszke
Thanks!
supakjk
apaszke
Aside from performance considerations, if the error is only being evaluated in terms of the output of a single module, aren’t there other (possibly negative) implications of updating the weights for components unused in the forward pass?
jablongo