I have a game with a square board in which the player moves around. They cannot move onto the last square that they were previously on, or out of bounds, but they can move around anywhere inside the square.
The game state is represented as a matrix. We mark the edge of the play area with a magic number and the player with a magic number, as well as the player’s previous space. There are a few other magic numbers for items in the play area that are not used (yet). The player starts at a random location.
We scale all of these between 0 and 1, then flatten it as input to the agent. The reward is 1 for making a proper step, and -1 for stepping out of boards or on it’s previous location (the game resets in this case). There are four output moves corresponding to each direction. This agent uses vanilla policy gradients based on the reinforce.py example in the PyTorch repository, but we’ve expanded the network size to be 2 hidden layers with 128 nodes.
In nearly all cases, the agent seems to very quickly converge on moving in only a single or two directions until it hits out of bounds. I think this is a local minima, since by going only in a corner direction or single direction it never accidentally moves onto its previous space. But this is clearly not optimal since it does not learn to avoid out of bounds.
I’ve tried using convolutional layers, various network sizes, various learning parameters, verifying the input data, using an actor-critic model, encouraging exploration by randomly forcing a move with certain probability, different rewards, slightly different game mechanics, and a range of other things but I must be missing something! Everything I’ve tried results in the (believed) local minimum, with the exception of forcing a random move which also results in an exploding gradient and crash as it gets close to the local minimum (log probability of the forced move becomes huge as the probability approaches zero). Any advice for debugging this?