Thank you both for your answers.



 SpandanMadan:

More context would be helpful


My code is quite spread out in numerous files, so I could not include it; also it was hard to do a minimal reproducible example.



 albanD:

What version of pytorch are you using?


I’m using version 0.2.0.



 albanD:

you modified a Tensor inplace during the forward pass


Thank you for suggesting this. It helped me track down the following operation. Here, I just want to modify a parameter inside a module:
self.my_param.data = other_tensor  # where my_param is a nn.Parameter
# other_tensor has a different size than self.my_param.data

Then I do a forward and a backward pass, without any problem. Then, I modify the parameter again. I do a forward pass, then a backward pass, and there I have the RuntimeError. The reason is that .grad is allocated on the first backward pass. The second time I change .data, the size of .data becomes different from the size of .grad.
Thus the error: RuntimeError: invalid argument 3: sizes do not match
My solution to this was to replace the parameter completely instead of assigning to .data:
self.my_param = nn.Parameter(other_tensor)

Now everything works.
Should this be prevented?
If we want to prevent this from happening, there are many options.

Should it be forbidden to assign a new tensor to my_param.data ?
Should it be forbidden to assign a tensor of a different size to my_param.data ?
Should my_param.grad be reassigned automatically when the size of .data changes?
Should the sizes of all .grad be matched to their .data when calling optimizer.zero_grad() ?

Options 2 and 4 make sense to me and seem not to complicated to implement. But in any case, it will add overhead.
Any thoughts?