Yes.
So in gross simplification, we want to optimize E(R(a)), the expected (E) reward (R) that is a function of our action (a). This is similar to wishing to optimize the loss (except it is minimization) where we want to optimize E(L(p)) the expected (E) loss (L) of a prediction (p) in supervised learning.
Now in supervised learning, the loss and prediction is typically framed as something continuous, either a distance (in regression) or a score/probability (in classification) and you can differentiate E(L(p)) by p. In the RL methods tackled with REINFORCE-type methods, you typically have discrete actions are discrete and thus are non-differentiable. But you do have the ability to move the distribution of the actions, thus changing the “density” implicit in the expectation E. That is then differentiable.
Similar things are done elsewhere, too, e.g. in Mathematical Finance if you do Monte-Carlo pricing and want to estimate sensitivities, you can sometimes do path-wise derivatives (i.e. differentiate the integrand in the expectation) and sometimes you have to use the “Likelihood ratio method” (which is differentiating the density to get a derivative of the expectation). From the top of my head Glasserman, Monte Carlo Methods in Financial Engineering, treats this in chapter 7.
Best regards
Thomas