Deep reinforcement learning has successfully solved a number of challenging control tasks. However, learned policies typically have difficulty generalizing to novel environments. We propose an algorithm for learning programmatic state machine policies that can capture repeating behaviors. By doing so, they have the ability to generalize to instances requiring an arbitrary number of repetitions, a property we call inductive generalization. However, state machine policies are hard to learn since they consist of a combination of continuous and discrete structure. We propose a learning framework called adaptive teaching, which learns a state machine policy by imitating a teacher; in contrast to traditional imitation learning, our teacher adaptively updates itself based on the structure of the student. We show how our algorithm can be used to learn policies that inductively generalize to novel environments, whereas traditional neural network policies fail to do so.
# Summary
This paper proposes a technique for synthesis of state machine policies for a simple continuous agent, with a goal of
them being generalizable to out-of-distribution test conditions. The agent is modeled as a state machine with constant
or proportional actions applied in each node (regime), and switching triggers between the regimes represented as
length-2 boolean conditions on the observations. The technique is evaluated on 7 classic control environments, and found
to outperform pure-RL baselines under "test" conditions in most of them.
# Review
I am not an expert in RL-based control, although I'm familiar with the recent literature that applies formal methods to
these domains. I find the studied settings valuable albeit fairly limited, but the paper's method undeniably shows
noticeable improvement on these settings. Inductive generalization is an important problem, and the authors' approach of
limiting the agent structure to a particular class of state-machine policies is a reasonable solution strategy.
That said, the complexity of synthesizing a state machine policy clearly caused the authors to limit their supported
action and condition spaces considerably (Figure 6). That, I'm assuming, limits the set of applicable control
environments where optimization is still feasible. The authors don't provide any analysis of complexity or empirical
runtime of the optimization process. Breaking it down for each benchmark would allow me to appreciate the optimization
framework in Section 4 much more. As it stands, Section 4 describes a complex optimization process with many moving
parts, some of which are approximated (q* and p(τ|π,x₀)) or computed via EM iteration until convergence (π*). It is hard
to appreciate all this complexity without knowing where the challenges manifest on specific examples.
Section 4.2 needs an example, to link it to the introductory example in Figure 1. The "loop-free" policies of the
teacher are, in programmatic terms, _traces_ of the desired state machine execution (if I understand correctly), but
this is not obvious from just the formal definition.
The EM optimization for the student policy makes significant assumptions on the action/condition grammars. Namely, the
algorithm iterates over every possible discrete "sketch" of every program, and then numerically optimizes its continuous
parameters (Appendix A). When the action/condition grammars grow, the number of possible programs there increases
combinatorially. Is there a way to adapt the optimization process to handle more complex grammars, possibly with
decomposition of the problem following the program structure?
Section 5 needs a bit more details on the Direct-Opt baseline. It's unclear how the whole state machine policy (which
includes both discrete and continuous parts) is learned end-to-end via numerical optimization. Granted, the baseline
performs terribly, but would be great to understand how it models the learning problem in order to appreciate why it's
terrible.
Why were the "Acrobot" and "Mountain car" benchmarks removed from the main presentation of results?