This was a long time ago and I forgot most of this, but I will say the best I could get going was based off of GitHub - ikostrikov/pytorch-a3c: PyTorch implementation of Asynchronous Advantage Actor Critic (A3C) from "Asynchronous Methods for Deep Reinforcement Learning".. Hope thats helpful