No. It seems to have the right dimensions. I will try to explain in words what I’m doing and then try to generate a minimal example. I am working on a minimal example, but I couldn’t get it to fail in the exact same way yet.
I’m learning a model for sequence tagging that predicts tags sequentially. The model uses word embeddings, and tag embeddings for the previous predictions. The main aspect is that the scores used for prediction are computed cumulatively. The training losses are computed based on these scores. The scores for the first prediction and second prediction are $s _1 = s( x, \emptyset ; w) $ and  $s _2 = s( x, \hat y _1 ; w) + s _1$, respectively. $x$ is the sentence and $\hat y _1$ is the prediction induced by $s _1$. The loss $\ell$ is computed based on the scores (and some gold data that I’m omitting because it is not important here). Doing backward on $\ell(s _1)$ works, but backward on $\ell(s _2)$ fails. I do a sequence of backward calls and take a step only at the end of the sequence.
I tried various things, but still got the same error: updating with each predicted tag; doing retain_variables=True; accumulating all losses and doing backward only at the end. I suppose that this is due to me not fully understanding what you can and cannot do with these computational graphs. What is the best way of training with cumulative scores on PyTorch?
I’m running on the CPU and using version 0.1.12_2.
Thanks a lot for looking into this.