The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample efficient. Our main contribution is SECRET, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.
This paper proposes to consider the problem of transfer in the context of sequential decision-making -- in particular reinforcement learning -- from the view-point of learning transferable credit assignment capability. They hypothesize that by learning how to assign credit, structural invariants can be learned which the agent can exploit to assign credit effectively and thus learn more efficiently in new environments (be it in-domain or out-of-domain). They pose the credit assignment problem as learning to predict (sparse) rewards at the end of sub-trajectories, finding the extent to which past state-action pairs appear to be responsible for these rewards (by means of the reward-prediction training), and creating a dense reward function via reward shaping (such that the set of optimal policies does not change). This is appealing as no modifications are needed to the RL algorithm/architecture. To examine their hypothesis, they created a method, called SECRET, based on self-attention and supervised learning to train credit assignment capability offline: sample many trajectories (often a mixture of expert and non-expert ones) from the source distribution, train a self-attentive seq2seq model to predict the rewards in these trajectories offline. Once this model is trained, they apply this model to a relatively small set of sampled trajectories from the target distribution and obtain the attention weights. Then, they use these attention weights as a proxy for credit assignment and, thus, use them to form a reward redistribution function. In their experiments, they show that the average attention weights actually signal the state-actions at which the future reward is triggered. They also show in their experiments that SECRET improves learning performance on in-domain transfer learning (larger mazes), as well as an out-of-domain case (with modified dynamics).             
Overall, this paper proposes an interesting general avenue for research in transfer learning in RL. Regarding the proof-of-concept method and experiments, I need some clarifications. Given these clarifications in the authors' response, I would be willing to increase my score.
1. Regarding this statement on breaking Markov property: "hide a certain amount of information from states and break the Markov assumption". 
(i) It is unclear to me what this "certain amount" would need to be in general. I believe this would require domain-specific knowledge to know what can be removed to break Markov-ness while not introducing significant state-aliasing (which could hinder the agent's learning). 
(ii) Does any extent of partial-observability warrant that the success in reward-prediction would mean that we have a valid credit assignment model? I feel like this is not generally true, in which case I question the statement on p.3: "Note that in POMDPs, this is unnecessary since the observations agents get from the environment are incomplete by construction."      
(iii) Regarding generality, the fact that states need to be (manually) preprocessed seems to me like a downside of this approach. Can you see any way around this?    
2. In p.4, this is mentioned: "In POMDPs, we recover an approximate state from the observation...".
I do not see how this is done in the DMLab experiments. If this is done manually, and not trained, then I think it should be clearly stated in the main text. I think the 2nd paragraph of Sec. A.4 is stating that extra information about the state was utilized, and not approximated via a trained model to recover the states (i.e., no s^=h^-1(o) was used)? 
3. What is the observation type of Vanilla RL in the out-of-domain experiments? Is it also observing its local-view (similar partial observability as SECRET) or does it have access to the full state? I would argue that it is important that the performance of Vanilla RL with partial observation is reported. Including both cases could also be beneficial. 
4. Fig.3 shows attention weights on held-out environments from an identical distribution as the source (i.e., in-domain). 
I would like to see how well the attention signal works when the target distribution differs from the source. Is there a reason why this is not demonstrated?  
5. Not sure about specific definitions of sub-trajectory and trajectory in the paper: 
(i) What constitutes a sub-trajectory (as opposed to a trajectory) in the context of this paper?
(ii) Are the lengths of the sub-trajectories or trajectories fixed?
6. Why do the attention weights not sum to 1 in Fig.3?
7. Could you clarify the role of positional encoding and how it is done?
Minor comments:
1. M is used to denote both MDP and causal map. 
2. Explicitly defining d_i in p.3 should improve clarity.  
3. Using 40k and 10k trajectories of interactions to train the credit-assignment model (on Triggers and DMLab domains, respectively) seems quite demanding, which seems somewhat unrealistic to deem useful for application to robotics perhaps?