That’s an interesting problem. Of course, the algorithm REINFORCE  as implemented in the official example does work on the cliff game. But your code will never learn anything for two reasons:


Your implementation of the cliff is not 100% exactly the game from the book (and quite a complex implementation for a simple grid-world).


The model (not the algorithm) you are using is not adapted to the type of inputs.


First, in the cliff game, the only terminal state is the goal. If you reach the cliff, you have a transition that leads you back to the starting point with reward -100. This detail is important, because if you break episodes at cliffs, you may never explore enough and quickly learn to stay around the start point. Here is a simple way to implement the true cliff game as a gym environment:
class CliffWalking(object):
    def __init__(self):
        self.R = -np.ones((4, 12))
        self.R[-1, -1] = 0
        self.R[-1, 1:-1] = -100
        print(self.R)

    def reset(self):
        self.x = 3
        self.y = 0
        obs = self.x + 4*self.y
        return obs

    def step(self, action):
        if action==0 and self.x<3:
            self.x += 1
        if action==1 and self.x>0:
            self.x -= 1
        if action==2 and self.y<11:
            self.y += 1
        if action==3 and self.y>0:
            self.y -= 1

        # get reward:
        reward = self.R[self.x, self.y]

        done = False
        # if goal, done= True:
        if self.x==3 and self.y==11:
            done = True

        # reset if goal or cliff:
        if self.x==3:
            self.y = 0

        obs = self.x + 4*self.y
        info = (action, self.x, self.y, reward)
        return obs, reward, done, info

Now, the inputs (states) are a position in the grid (in your code, it’s x-y coordinate, in my it’s simply x + 4y, the coordinate in a flat version of the grid). For a neural network, such integer inputs are impossible to handle. A neural network expects floating values, for ex between -1 and 1 or between 0 and 1. One solution would be to use an embedding: for ex,  take a vector of length 12*4 with zero everywhere but a one at your position in the grid. But that’s very complicated for nothing and very hard to learn because of the number of parameters. A much simpler solution is to use a tabular model instead of a neural network:
class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.Q = nn.Parameter(torch.zeros(1, 4*12, 4))
        self.saved_log_probs = []
        self.rewards = []

    def forward(self, x):
        action_scores = self.Q[:,x,:]
        return F.softmax(action_scores, dim=1)

Then, you can chose long-enough episodes (1000 steps) so you ensure that your algorithm will reach the goal by chance at least once:
def main():
    for i_episode in range(500):
        state = env.reset()
        sum_reward = 0
        for t in range(1000):
            action = select_action(state, i_episode)
            state, reward, done, info = env.step(action)
            sum_reward += reward
            if args.render:
                env.render()
            policy.rewards.append(reward)
            if done:
                break

        running_reward = sum_reward
        finish_episode()
        if i_episode % args.log_interval == 0:
            print('Episode {}\tSum rewards: {:.2f}'.format(
                i_episode, running_reward))

Now it should work quite well, and using good learning rate (0.5 with Adam descent) it learns in ~ 100 episodes a near-optimal solution using 16 action-steps (the opimal is 13, but is “unsafe”, as described in the book).