Solved by 111119 in post #2 
              
              
                I solve it. 
The problem is in computational method. 
I used softmax at my actor-network’s final activation function. 
After that, The output(softmax_probability) is used to calculate log(torch.log10) 
But I changed softmax to log_softmax function and then success.