Do you refer to MixtureSameFamily? That one contains a nested “Categorial” distribution object, with non differentiable distribution parameters.
It is possible (though not trivial) to train Categorical with sampling - docs describe REINFORCE / score function. This issue is orthogonal to using gradient descent to train GMMs in general, or “unitary constraint” (that is handled by using Categorical distribution [class])