I have a RNN and after every batch, I have the option of either detaching the hidden states or re-initializing them. It’s not clear to me which one should I choose. If the batches are independent, should you just re-initialize them (like with all zeros) or should you pass the hidden state’s data to the next batch (but call detach so you don’t backprop through the entire dataset).
Also, as as a sanity check:  if the batches were dependent you would call detach if you had to choose between re-initializing or detaching?