Hello again. First of all, thank you for the contribution @KFrank.
I didn’t reply @KFrank 's answer immediately since I knew that it would take me some time to implement all of that. I think I have finished the code and guess what? It is not working   I do understand the reason but I don’t know the solution. So here I am once more.

This time I’m gonna use a drawing which I hope will explain the condition far more easily. I’m trying to do Inverse RL:

image705×410 28.3 KB

I don’t have the actual rewards(R). That’s why, I cannot simply use mse_loss(R, R_est). I calculate the distance between π and π_est somehow and want to use this value as the error signal for the backpropagation.
I believe the following 2 lines sum up my issue:
loss = torch.nn.functional.mse_loss(emp_fc, esvc)
loss.backward()

I get this error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor []], which is output 0 of SelectBackward, is at version 200; expected version 199 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Can anybody help me solve this problem? I don’t know how can I avoid in-place operations while I have soooo many of them. Nevertheless, I have to backpropagate that value.