I am also confused and I was going to post a similar question.
My guess is that you should run everything in train mode or only the action selection should be with model.eval(). I think training might become unstable if you use results from one model in eval and another one in train mode.