In the naive REINFORCE method (which is used in the example), we use \Delta log \pi_\theta v(t) to do updating. Just forget cross-entropy loss. PyTorch provide inforce() method for Variable to bind the corresponding v(t) in the formula. You can try following code for checking:
x = Variable(torch.Tensor([0.1, 0.9]), requires_grad=True)
y = torch.multinomial(x, 1)  # here, make sure y.data[0] = 1
r = torch.Tensor([2]).float()
y.inforce(r)
y.backward()
print x.grad.data

Which will give you the output of [0.0, -2.2]. And -2.2 is exactly 1.0/0.9 * -v(t) (Because dlogp = 1/p)