Hi Stijn,
I’m not really familiar with RL and I’m not sure to see all possible pitfalls in this approach.
A “safe” approach could be to detach the target before calculating the loss.
This would make sure to only backpropagate using the first input parameter to the criterion.
Would that work for your approach?