I solve it.
The problem is in computational method.
I used softmax at my actor-network’s final activation function.
After that, The output(softmax_probability) is used to calculate log(torch.log10)
But I changed softmax to log_softmax function and then success.