For each episode, after each action, when I receive the new state, the reward and termination boolean, I forward the new state. Basically, here’s what I do:
mask = 0. if done else 1.
new_state_value = value(new_state).detach()*mask
v_n_mem.append(new_state_value)