This paper proposes a new method to drastically speed up deep reinforcement learning (deep RL) training for problems that have the property of state-action permissibility (SAP). Two types of permissibility are defined under SAP. The first type says that after an action a t is performed in a state s t and the agent reaches the new state s t+1 , the agent can decide whether the action a t is permissible or not permissible in state s t . The second type says that even without performing the action a t in state s t , the agent can already decide whether a t is permissible or not in s t . An action is not permissible in a state if the action can never lead to an optimal solution and thus should not be tried. We incorporate the proposed SAP property into two state-of-the-art deep RL algorithms to guide their state-action exploration. Results show that the SAP guidance can markedly speed up training.
This paper proposed the concept of state-action permissibility (SAP). Given a user-defined type 1 SAP function, the algorithm learns a classifier to predict whether an action at a given state is permissible or not. Based on this prediction, the reinforcement learning (RL) algorithms can limit the exploration only to the permissible actions, and thus greatly reduce the cost of learning. The proposed algorithms are tested on two simple tasks, both of which have the same flavor of following a predefined track.
Although the results of the experiments show that SAP helps to speed up RL, I think that the application of SAC is very narrow. It is extremely difficult to define an AP1 function in general. For example, for most of the OpenAI gym environments (such as half-cheetah, ants or humanoid), it is not clear to me how to manually define an AP1 function. It would be more convincing if the paper can apply the proposed techniques to some of the benchmark OpenAI gym environments.
Even for the lane following task described in the paper, the AP1 function in eq. 5 is limited and eliminates many good solutions. It constrains that the action should not lead to more deviations to the center line in the next time step. This greedy constraint will not work in more interesting driving scenarios. For example in a sharp turn, if the curvature of the lane is too large for the car to follow, a common strategy (that can be learned by vanilla RL algorithms) is to first drive to the outer side of the lane before the turn, cut to the inner side at the turn and exit the turn to the outer side. This optimal solution to negotiate a tight turn is completely eliminated by the user-defined AP1 function (eq. 5).
The idea of AP1 is somewhat contradictory to the philosophy of reinforcement learning. AP1 is a greedy decision based on the next step while RL optimizes for the accumulated reward over many steps. RL allows taking an action that will sacrifice the immediate reward (e.g. deviate from the center line of a lane) in the next step but can accumulated more reward in the long run (successfully drive along a tight turn). In most of cases, by looking at the next state, it is just not possible to predict whether a specific action cannot lead to the optimal long-term reward (SAP).
For the above reasons, I think that the application of SAP would be very narrow, especially for reinforcement learning. I would not recommend accepting this paper at this time.