My wild guess at the moment is that because the object we use to accumulate gradients is only a weak pointer into the Tensor, it is not always cleared before you run the next backward. In that case, it is re-used and fails because the size changed.
@ezyang do we expect the .set_() (and .resize_()) operation to reset the grad_accumulator_ ?