The thing that seems to be typically done with this under the name “through truncated time” aka BPTT is to “detach everything” that is carried over between the backwards.
Part of that is the exploding/vanishing gradient problem and the other part is the administrative component we are discussing here.