Just a minor slightly off-topic warning:
One thing I stumbled upon using REINFORCE, is that, if you run it often enough, you will by chance pick a highly unlikely action at some point in time. Since it uses the log of this highly unlikely action the gradient explodes end might entirely mess up your agent, especially when using a momentum based optimizer. So you should definitely use a gradient clipping to avoid that (sadly it took my quite a while to figure that out…)