Machine Learning, Optimization, Gradient Descent, Reinforcement Learning.
back prop through dz/dπ since the sampling process is non-differentiable. With Gumbel softmax, we have reparameterized our sampling procedure such that dy/dπ is differentiable. For differentiable environments, this lets us backprop everything from the reward to the parameters of the model. But