Solving tasks in open environments has been one of the long-time pursuits of reinforcement learning research. We propose that data confusion is the core underlying problem. Although there exist methods that implicitly alleviate it from different perspectives, we argue that their solutions are based on task-specific prior knowledge that is constrained to certain kinds of tasks and lacks theoretical guarantees. In this paper, Subjective Reinforcement Learning Framework is proposed to state the problem from a broader and systematic view, and subjective policy is proposed to represent existing related algorithms in general. Theoretical analysis is given about the conditions for the superiority of a subjective policy, and the relationship between model complexity and the overall performance. Results are further applied as guidance for algorithm design without task-specific prior knowledge.
This paper introduces a new framework for reinforcement learning (named subjective reinforcement learning) which aims to resolve some of the inherent problems with RL in open environments.  The authors posit that one problem with RL in open settings is “data confusion”, which they describe as being situations where there are external factors (e.g. timeframe) that affect the action space differently.  They propose a “subjective reinforcement learning framework” which, as I understand it, can be described as an ensemble of traditional MDP’s subject to external factors k.  The paper evaluates how this subjective policy compares to traditional MDP’s in terms of theoretical bounds on performance. 
Although this learning framework seems like a potentially interesting future research direction, I tend to lean towards rejection for the reasons that: (1) the theoretical analysis is difficult to follow and there is sometimes a lack of clarity throughout the paper, (2) it isn’t very clear how easy this framework would be to implement aside from the theoretical guarantees and there aren’t any experiments or proofs of concept that would demonstrate the feasibility or practicality of the proposed framework in a real scenario, (3) the paper would benefit from more discussion of how their work differs from related techniques (like hierarchical RL, various forms of meta-learning, etc.).
Here are some more concrete points:
(1) The feasibility of this framework in a real-world scenario seems a bit hard to imagine and a strong use-case or proof-of-concept would be very helpful.  I liked that this paper provided some theoretical analysis for guiding the design of these systems.  However, it seems like these claims would also benefit from detailed empirical analysis and experiments.  Without empirical results, I feel a bit skeptical about how straightforward it would be to implement such a system or whether it would really be significantly useful in practice.  Similarly, though there are theory-based suggestions for how to optimally design such a system, it might be difficult to implement this system with optimal hyperparameters in a real-world use-case and the challenges in doing so are not really addressed.
(2) In spite of claims that this method is able to be trained without domain knowledge, it seems like domain knowledge would still be necessary for things like determining what external information (K) is available and necessary, determining the appropriate N_S, etc.  It may be helpful for the authors to explain a bit more about how these things can be determined in a truly agnostic way.
(3) It seems like there should be more discussion of the difference from hierarchical reinforcement learning.  In practice, hierarchical RL also can be used in similar ways to what’s described here.  As the authors point out, hierarchical RL is not necessarily splitting into submodels that handle data confusion problems, it seems like that is a constraint that could be added into a hierarchical framework’s design.
(4)  I appreciate that the authors provide detailed theoretical analysis, but it can sometimes be confusing and difficult to follow.  I had some trouble evaluating the correctness of several of the proofs.  It may benefit from re-writing with more concise definitions of all of the variables and more clearly stated assumptions about observable information.  Here are some points of confusion for me:
    - It seems like certain letters (eg. A or K or N_s vs N_d) are being overloaded as variable names with different fonts.  I realize that this is somewhat unavoidable, but I would recommend that the authors try to disentangle the naming a bit for improved clarity.
    - In theorem 1, you stated "the number of all possible data samples tends to infinity when the total number of samples N_d approaches infinity".  This proposition seemed confusingly worded to me.  Maybe I am misunderstanding the wording, but it seems like possibly a tautology?
    - I’m not sure I understand what is meant by “fake subjective policies” in Theorem 1.  Could you explain what is meant by that and the intuition here a bit more?
    - It’s still unclear to me how K (that is, the external information) is being collected at any given timeframe.  Are you assuming that the necessary types of external information corresponding to K (your examples are ‘state history, out-of-MDP task encodings, samples from related tasks, etc.’) have been pre-determined?  If so, how could the most-appropriate type of external information be chosen in a practical way?
I also noticed a few (very minor) grammar errors that authors may want to fix, though they did not affect my review:
- page 1: "tasks in open environments poses difficulties" --> "tasks in open environments pose difficulties"
- page 1: "Problem is that both..." --> "The problem is that both..."
- page 2: "we propose a novel framework named as Subjective reinforcement" --> "we propose a novel framework named Subjective reinforcement"
- page 4: "should contain no data confusion" --> "should not contain data confusions"