Prob not helping, but when you say the reward * log_prob, I think you mean the return * log_prob. They are different. Reward is what is given at each step of the episode. Return is the expected cumulative reward over the entire episode from the time t onward. In Sutton’s book the return is represented by ‘G’ in the pseudocode on page 271.