I am following the REINFORCE implementation on cartpole from here:

  
      github.com
  
  
    PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter09/02_cartpole_reinforce.py 3
#!/usr/bin/env python3
import gym
import ptan
import numpy as np
from tensorboardX import SummaryWriter

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

GAMMA = 0.99
LEARNING_RATE = 0.01
EPISODES_TO_TRAIN = 4


class PGN(nn.Module):
    def __init__(self, input_size, n_actions):
        super(PGN, self).__init__()



  This file has been truncated. show original

  
  
    
    
  
  


The network does a very good job in solving the cartpole, but when i change line 98 from
log_prob_actions_v = batch_qvals_v * log_prob_v[range(len(batch_states)), batch_actions_t]


to
log_prob_actions_v = batch_qvals_v * log_prob_v[:, batch_actions_t]

the code works, but mysteriously, the algorithm does not converge at all…why?
Also can somebody show me how to implement this without using ptan? I am not getting how the library is working… It would help a lot…
Thanks in advancee