According to my observation, nan could be caused by a lot of things: inappropriately designed reward, the model itself, invalid input data etc.
I would suggest you print “.grad” attribute for every parameter out, along with their comming source (worker rank), normally before your nan occurs you will see several ridiculously large gradients like 1e8 or 1e11.