Ah, can backpropagate it in one go. Entropy normalization still adds a lot of time, but not as bad as backpropping twice:
opt.zero_grad()
entr_loss = - entropy_reg * entropy
a.reinforce(reward)
autograd.backward([a, entr_loss], [None, None])
opt.step()