The problem is that the samples from the categorical distribution are discrete, so there is no gradient to compute. This is in contrast to the Gaussian where you can write X = Z * sigma + mu with Z ~ N(0,1) to get a N(mu, sigma)-distributed variable  (the reparametrization trick in some circles).
A common way around this is to not sample, but compute the loss for all categories, weighted with their respective probabilities. This then allows to compute gradients w.r.t. the probabilities.
Best regards
Thomas