Active Exploration for Learning Symbolic Representations
We introduce an online active exploration algorithm for data-efficiently learning an abstract symbolic model of an environment. Our algorithm is divided into two parts: the first part quickly generates an intermediate Bayesian symbolic model from the data that the agent has collected so far, which the agent can then use along with the second part to guide its future exploration towards regions of the state space that the model is uncertain about. We show that our algorithm outperforms random and greedy exploration policies on two different computer game domains. The first domain is an Asteroids-inspired game with complex dynamics but basic logical structure. The second is the Treasure Game, with simpler dynamics but more complex logical structure.
Key question asked in the paper: Can an agent actively explore to build a probabilistic symbolic model of the environment from a given set of option and state abstractions? And does this help make future exploration easy via MCTS? This paper proposes a knowledge representation framework for addressing these questions.
Proposal: A known continuous state space and discrete set of options (restricted to subgoal options) is given. An options model is treated a semi-MDP process. A plan is defined as a sequence of options. The idea of a probabilistic symbol is invoked from earlier work to refer to a distribution over infinitely many continuous low-level states. The idea of state masks is introduced to find independent factors of variations. Then each abstract subgoal option is defined as a policy that leads to a subgoal for the masked states, for e.g. opening a door. But since there could be many doors in the environment, the idea of a partitioned abstract subgoal option is proposed to bind together subgoal options for each instance of a door. The agent then uses these partitioned abstract subgoal options during exploration. Optimal exploration then is defined as finding a policy via MCTS that leads to greatest reduction of uncertainty over distributions over proposed options symbolic model.
Feedback:
- There are many ideas proposed in this paper. This makes it hard to clearly communicate the contributions. This work builds on previous work from Konidaris et al. Is the novel contribution in the MCTS algorithm? The contributions should be made more explicit for unfamiliar readers. 
- It is difficult to keep track of what's going on in sec 3. I would additionally make alg1 more self-inclusive and use one of the working examples (asteroid or maze) to explain things. 
- Discuss assumption of restriction to subgoal option. What kind of behaviors does this framework not permit? discuss this in the paper
- Another restrictive assumption: "We use factors to reduce the number of potential masks, i.e. we assume that if state variables i and j always change together in the observations, then this will always occur. An example of a factor could be the (x, y, z) position of your keys, because they are almost never moved along only one axis". What are the implications of this assumptions on the possible range of behaviors?
- The experimental setup needs to be developed more to answer, visualize and discuss a few interesting questions: (1) how does the symbol model change with active exploration? (2) visualize the space of policies at various intermediate epochs. Without this it is hard to get an intuition for the ideas and ways to expand them in the future.
- I believe this is a very important line of work. However, clarity and articulation in the writing/experiments remains my main concern for a clear accept. Since some of the abstractions are fairly new, the authors also need to clearly discuss the implication of all their underlying assumptions.