I have a custom Module with certain parameters. One of them is a ModuleDict() which has 2 parameters(Linear modules) for key values say ‘a’ and ‘b’ . Depending on the training example(one minibatch) provided, only one of these parameters should be used per minibatch in the forward pass - they key value for the moduledict is decided by the minibatch. I might be seriously misunderstanding pyTorch and NN’s in general here, but is it not the case that if a certain parameter is not used at all in the forward pass, its gradient(param.grad.data.sum()) should be zero AND it should not change in value at all for that minibatch? The gradient is zero, however the value of both the parameters(in this case, the weights and biases of the Linear modules) change every minibatch - only one of them should change.
A side question - Is there a good way to visualize the history a variable while debugging in python? Preferably not inside a jupyter notebook.