Hi,
I was trying to implement the REINFORCE algorithm from scratch and somehow the policy was not improving at all. I will cut the details and will post the culprit part of the codes (mine vs code from pytorch examples). Can someone please tell me what is the difference between the codes below. The seeds and network is same
Code That Does Not Work

class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.network = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 2), nn.Softmax())
        self.log_probs = []
        
self.rewards = []
    
def forward(self, state):
        temp_state = torch.from_numpy(state).float().unsqueeze(0)
        return self.network(temp_state)


    def train(self):
        exp_return = 0
        returns = []
        #policy_loss = []
        for reward in self.rewards[::-1]:
            exp_return= reward + 0.99*exp_return
            returns.insert(0, exp_return)
        returns = torch.tensor(returns)
        policy_loss = Variable((returns * torch.tensor(self.log_probs)).sum(), requires_grad=True)
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()
        self.rewards = []
        self.log_probs = []


    def policy_add_reward(self, reward):
        self.rewards.append(reward)

    def select_action(self, state):
        probs = self(state)
        m = Categorical(probs)
        action = m.sample()
        self.log_probs.append(-m.log_prob(action))
        return action.item()


Code that Does Work
That can be found here 24
I have added a debugger and seen the policy_loss for both of the codes. Since the seed was same, the loss was consistent.