Well, so outside RL this outer expectation is usually covered by minibatching + the iteration. Just like Algorithm 2 in the reference seems to suggest that a single a_t, s_t+1 pair is sampled in line 7.
Of course, the usual thing is to do something that is vaguely linear in the gradient, so it might not be applicable.
What is the exact maths that you’re trying to translate?