Expanding Non Markovian Reward Decision Processes (NMRDP) into Markov Decision Processes (MDP) enables the use of state of the art Reinforcement Learning (RL) techniques to identify optimal policies. In this paper an approach to exploring NMRDPs and expanding them into MDPs, without the prior knowledge of the reward structure, is proposed. The non Markovianity of the reward function is disentangled under the assumption that sets of similar and dissimilar trajectory batches can be sampled. More precisely, within the same batch, measuring the similarity between any couple of trajectories is permitted, although comparing trajectories from different batches is not possible. A modified version of the triplet loss is optimised to construct a representation of the trajectories under which rewards become Markovian.
** Summary
The paper studies a specific class of Non-Markovian Reward Decision Processes (NMRDPs). In general, in NMRDP the dynamics is Markovian but the reward function may depend on the entire trajectory. The authors consider a sub-case where the trajectory can be mapped to a specific "task" and the reward function can be formalized as a mapping between the state-task pair to the reward. This greatly simplifies the problem that can be mapped onto an augmented MDP and solved using standard RL tools. The authors focus on the representation learning problem of the mapping between trajectories to an embedding of the task itself. In particular, they consider contrastive learning to find a representation that discriminate between trajectories associated to the same task and trajectories coming from different tasks. The resulting LSTM-based architecture is then evaluated on a grid-world synthetic problem and on GPS trajectories from tourists in the city of Salzburg.
** Overall evaluation
My main concerns about the paper is that the exact objective of the setting and the representation learning is not that clear and the empirical validation is limited to a qualitative assessment of the representation learning with no down-stream task. More in detail.
1- The assumption of task-dependent reward functions is very sensible. Yet I wonder to which extent it overlaps with literature in multi-task and meta-learning, or more in general with embedding of time-series (the trajectory in your case). It would be useful to frame/compare at the conceptual level the similarities and differences of the proposed setting with those fields.
2- At training time, the PK dataset is somehow supervised, as it is possible to know which trajectories are associated to the same task. At test time, the learned f_theta recursively maps trajectories to tasks. As such, it seems like it could effectively detect changes into the task itself by tracking how the trajectory evolves (when the task function is unknown). This aspect is never really evaluated in the empirical section, but it would be one of the most interesting uses of the learned representation.
3- If the task function T is known, it means that standard RL techniques can be used to solve MDP N. The actual advantage of using a specific function to embed the task to a space in Re^d is never really explained in the paper. Does it make solving N simpler or more effective than using a simple encoding for the task? No evaluation is available in this sense.
4- The empirical evaluation is limited to qualitative analysis of the representation learned in the problem. Yet, there is no clear support that the representation is good/useful in a more quantitative way (e.g., by actually solving the RL part or in identifying quickly the current task).
Some more specific questions/comments
1- Some of the notation is a bit redundant. For instance, phi is mapping from H to Re^d, while f is a mapping from trajectories to Re^d, but in the end they are doing the same thing, as H itself can be obtained as a mapping from trajectories to tasks through T. 
2- "Using RL techniques, finding an optimal policy pi* in the equivalent MDP \hat M is possible and by extension pi* is optimal for N". This passage is not fully clear, if phi is introducing some form of approximation, then pi* may no longer be optimal for the original MDP N. On the other hand, if phi is not approximating but "just" changing the representation from H to Re^d, then it is not clear what is the interest of it.
3- The assumption that similar trajectories can be identified is somehow strong. It would be good to have a more thorough support for it.
4- It is not fully clear what L_BH^local is indeed a local loss. It seems like you are simply using the mapping from the whole trajectory. Is this why it is called local?
5- While I appreciate that the introduction is sketching many different scenarios to support the models studied in the paper, in the end they mostly lack of depth and they rather give a confusing impression instead of clarifying in a compelling way what is the problem studied in the paper. I suggest you rather pick one single scenario with a good level of detail to provide a more solid support to the paper.
** Minor comments
1- You often use "he" to refer to the agent. It would be better to use "it" or "she".