Humans can learn task-agnostic priors from interactive experience and utilize the priors for novel tasks without any finetuning. In this paper, we propose ScoringAggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions with sparse reward and then plan on unseen tasks in zero-shot condition. The framework finds a neural score function for local regional state and action pairs that can be aggregated to approximate the quality of a full trajectory; moreover, a dynamics model that is learned with self-supervision can be incorporated for planning. Many previous works that leverage interactive data for policy learning either need massive on-policy environmental interactions or assume access to expert data while we can achieve the similar goal with pure off-policy imperfect data. Instantiating our framework results in a generalizable policy to unseen tasks. Experiments demonstrate that the proposed method can outperform baseline methods on a wide range of applications including gridworld, robotics tasks and video games.
The paper proposes a framework (Scoring-Aggregating-Planning (SAP)) for learning task-agnostic priors that allow generalization to new tasks without finetuning. The motivation for this is very clear - humans can perform much better than machines in zero-shot conditions because humans have learned priors about objects, semantics, physics, etc. This is achieved by learning a scoring function based on the final reward and a self-supervised learned dynamics model.
Overall, the paper is very clear and easy to follow.
The presented task is realistic and important, and the paper seems to address it in a reasonable approach. 
However, the evaluation seems lacking to me - the evaluation convinced me that SAP works, but I am not convinced that it works better than existing approaches (see below), and especially did not convince me that it is better in the zero-shot test environment.
The (anonymized) website contains nice videos that support the submission.
Questions for the authors:
1. Page 3, 3rd paragraph of Section 3: the paper says that "The proposed formulation requires much less information and thus more realistic and feasible" - I agree that this is more realistic, but is it really more feasible? The requirement of much less information makes the proposed formulation much more sparse.
2. A basic assumption in the SAP framework is that a local region score is a sum of all the sub-regions. As phrased in the paper: "in the physical world, there is usually some level of rotational or transnational invariance". I'm not sure that this assumption makes sense neither in the Mario case or in other tasks, e.g., robotics. Doesn't it matter if you have a "turtle" right in front of you (which means that the turtle is going to hit you), or below you (which means that you are going hit the turtle)?
3. A question about the planning phase - page 5 says: "We select the action sequence that gives us the best-aggregated score and execute the first action". Do you select the entire sequence of actions in the new environment in advance? Can the agent observe the new state after every action, and decide on the next action based on the actual step that the action has reached, rather than on the state that was approximated in advance?
In other words - what happens if the first action in the new test environment yields an unexpected state, that was not predicted well by the dynamics model; does the agent continue on the initial planned trajectory (that ignores the "surprise"), or does it compute its next action based on the unexpected state?
 
4. Experiments: in Gridworld and Mario - are there any stronger baselines in the literature, or reductions of known baselines to the zero-shot scenario? Are the chosen "Human Priors", BC-random and BC-SAP just strawmen? 
Since the main goal of this paper is the zero-shot task, what would convince me is a state-of-the-art model that does possibly *better than SAP on the training level*, but *worse than SAP in generalizing to the new level*. Additionally, are there other baselines that specifically address the zero-shot task in the literature?
Minor (did not impact score):
Page 2, 1st paragraph: "... we show that how an intelligent agent"...
Page 3, 3rd paragraph: "... in model-free RL problem" - missing an "a" or "problem*s*"?
Page 3, 3rd paragraph: ". Model based method ..." - missing an "a" as well?
Page 4, 1st paragraph:: "... utilizing the to get the ..."
Page 4, last row: missing a dot after the loss equation, before the word "In".
Page 7, Table 1: "BC-random" is called "BC-data" in the text. Aren't they the same thing?