Solved by albanD in post #4 
              
              
                Because in the init, you always do self.weight = nn.Parameters(foo). 
A workaound is to reset the current weight at the beginning of the forward with: 
del self.weight_current and del self.bias_current.