This was worded rather unfortunately.
You options are more or less

keep the state from previous batches but detach it,
set to zero at each batch (this obviously will not be attached to the last batch),

but you cannot

work with (undetached) state that is still connected to the last step: this will either lead to “trying to backward through the graph a second time” errors or, if you instruct PyTorch to keep the graph, memory exhaustion (and probably gradient blow-up, too).

Best regards
Thomas