Because in the init, you always do self.weight = nn.Parameters(foo).
A workaound is to reset the current weight at the beginning of the forward with:
del self.weight_current and del self.bias_current.