I think that, unfortunately, the program as described has both mathematical and PyTorch errors to make it quite a riddle what is meant.

The derivation of the second term of the loss function is not broken. As a result, I would expect to see logsumexp(theta + x, -1) for some appropriately shaped matrix in the loss function.
You would want to clamp the reference probabilities away from 0 to avoid -inf negative log likelihood.

theta = torch.zeros((1, 100), requires_grad = True).float() makes theta a non-leaf, which is broken. You really, absolutely need to avoid calling any function between constructor and assignment, so here (if it were not the default) one could use use dtype=torch.float in the constructor.
There are several other oddities / don't dos. e.g. you should only collect loss.item(), the order of optimizer steps is OK, but the usual way is zero_grad, backward,  step etc.

In summary, I would recommend to re-do the derivation unless Anthony has an update that makes the intention and code clearer.
Best regards
Thomas