I guess it can be backproped separately, using retain_grad=True? Something like:
x = F.softmax(x)
a = torch.multinomial(x)
entropy = - x * x.log()
entropy = entropy.sum(1).mean()

# later ...
opt.zero_grad()
(- entropy_lambda * entropy).backward(retain_graph=True)
a.reinforce(reward)
autograd.backward([a], [None])
opt.step()

?