My goal is to change the weights after the forward pass but before the backward.
If I use layer.weight = xxx then the backward pass ignores the new weights and uses the old ones. It seems that directly writing to layer.weight.data, which you advice against, is the only option that works.