What are you using for the state of your environment? It’s possible you have already accounted for this, but the agent needs some way to know where it was in the previous state of the environment. Otherwise, the only way it will be able to ensure it does not accidentally return to its previous location is to always go a single direction. If you have not accounted for this, try stacking the current observation with the previous observation for your state. Or a less computationally expensive state would be to subtract some portion of the previous observation from the current observation.
If you have a sufficient state and the problem continues to persist, it’s probably a bug in your code, but there is the chance that your hyperparameters need tuning. For something as simple as your game, good rules of thumb are using .99 for your reward discount factor, and .001-.0001 for your learning rate (this will depend on if you are averaging or summing the gradient terms). Also, a good piece of advice (from John Schulman) is to try increasing the amount of data you use per update. You probably won’t need anything more than 20,000 frames per update(?), but honestly this is one of those black magic things that is problem and implementation dependent.