I think that is already what I’m doing.
In the gist that I linked above, I just pass the agent to an evaluator function (the Test class) that runs in another process, from the main learner process that is responsible for exploiting the reward sent back from the evaluator. The evaluator function only uses the agent to make forward passes and gather the final reward at the end. I use a torch.no_grad in my evaluator function to make sure that no gradients are being calculated !
Is this what you mean ?