Solved by 111367 in post #4 
              
              
                Thanks albanD~ 
I find a way that it won’t pop up the issue by rearranging the code like the following bellow: 
value_loss = 0.5 * mse_loss(imaginated_values, lambda_target_values.detach())
value_optimizer.zero_grad()

action_loss = -1*(lambda_target_values.mean())
action_optimizer.zero_grad()

valu…