this is a way to do back prop, but the original is not incorrect, it depends on the problem. If you have a label at the end of a sequence then your solution is not applicable, but if you are update the parameters for a decoder, this is useful, but one potential issue (that i’m currently trying to figure out) is that won’t your hidden state only propagate the error from the last label, since the states are being detached? The gradient accumulation will be on the last hidden state therefore when you perform back prop on previous labels those will only be factored into the label’s respective timestep. Or so it seems to me, or will the older state still get gradient accumulation from previous label errors ? I might be assuming the wrong behavior about detach()