Hey Satchel,
Sorry for the long delay! I wanted to be sure to follow up on this. We implemented gradient averaging for the update, as well as a couple smaller tweaks like using an entropy-based loss component, and it did the trick! The agent seemed to move out of its local minima and start exploring actual strategies that led to a much longer episode length.
Thank you for your help!