Do you also get closer for the wrong ones? The reward suggests that one off is better than further away.
So here are the bits I changed:

The learning rate in the optimizer instantiation (I did not play much with it)

        self.optimizer = optim.Adam(self.model.parameters(), lr=3e-3)


in select_action the regularisation of the log

        log_prob = prob.clamp(min=1e-6).log()
        entropy = - (probs * probs.clamp(min=1e-6).log()).sum()


in update_parameters the “multiple independent episodes” reward (ideally, one would allow several multi-step episodes, but here is independence)

            R = self.rewards[i]


and finally I changed the inner loop to update parameters after 50 parameters (note the % 10 to only have inputs 0…9), put in 10001 outer loops and a diagnostic print every 500 outer loops

    reinforce = REINFORCE(128,1,10)
    for j in range(10001):
        for i in range(50):
            input = torch.FloatTensor([i%10]).unsqueeze(0)
            output, log_prob, entropy = reinforce.select_action(input)
            reward = -torch.abs(output.type(torch.FloatTensor) - input)
            if j%500==0 and i < 10:
                print (j, input[0,0], output[0,0], reward[0,0])
            reinforce.add_experience(reward, log_prob, entropy)
        reinforce.update_parameters()

.
With that I go from
0 0.0 0 -0.0
0 1.0 4 -3.0
0 2.0 8 -6.0
0 3.0 4 -1.0
0 4.0 4 -0.0
0 5.0 4 -1.0
0 6.0 8 -2.0
0 7.0 0 -7.0
0 8.0 4 -4.0
0 9.0 8 -1.0

to
10000 0.0 0 -0.0
10000 1.0 1 -0.0
10000 2.0 1 -1.0
10000 3.0 3 -0.0
10000 4.0 4 -0.0
10000 5.0 5 -0.0
10000 6.0 5 -1.0
10000 7.0 8 -1.0
10000 8.0 8 -0.0
10000 9.0 8 -1.0

so 60/40 correct and one off.
I must admit I didn’t try the usual tricks beyond the learning rate to try and make it work better (so reward scaling seems one, but I know little about RL theory at the moment and even less about practice except that Alex Irpan says its hard 5).
Best regards
Thomas