Trajectory representation learning for Multi-Task NMRDPs planning
Expanding Non Markovian Reward Decision Processes (NMRDP) into Markov Decision Processes (MDP) enables the use of state of the art Reinforcement Learning (RL) techniques to identify optimal policies. In this paper an approach to exploring NMRDPs and expanding them into MDPs, without the prior knowledge of the reward structure, is proposed. The non Markovianity of the reward function is disentangled under the assumption that sets of similar and dissimilar trajectory batches can be sampled. More precisely, within the same batch, measuring the similarity between any couple of trajectories is permitted, although comparing trajectories from different batches is not possible. A modified version of the triplet loss is optimised to construct a representation of the trajectories under which rewards become Markovian.
Short summary of paper: The paper investigates representation learning from trajectories in a framework which generalizes MDPs (NMRDPs), in which rewards are non-Markovian and follow a hidden process (this can be seen as a more structured, special case of a POMDP). The authors suggest learning a trajectory embedding by using a triplet loss, justified by a sufficient condition for learning an embedding which corresponds to the true task id. Overall I felt the paper fell a bit short of the standards for ICLR, based on subpar writing, lack of comparison to convincing baselines, and unclear applicability to more complex environments. Main issues: - The methodological sections of the paper seems conceptually accessible while being written in an overly mathematical, meandering fashion, using heavy, sometimes unclear notation, or referring to notation from other papers. As an example, mathcal{M}^* is first introduced by reference to another paper (the notation does not really need to be introduced until the mapping is made precise later in the paper); phi(	heta)_i^j(-1) is an unnecessarily heavy, unpleasant notation for the concept used. It was unclear if T is meant to represent the total trajectory length or a time index of a given trajectory (the notation would imply the former, the way it used, the latter). Similarly equation (6) is a very mathematical way to denote a relatively simple condition (which could be written more simply by comparing inf and sup distances between two trajectory embeddings corresponding to different or identical latent h). Again, equation (7) seems like a contrived way to write a relatively simple concept, and the simple proof that (6) implies (7) is not included. I feel this section could be improved by using precise but as simple notation as possible, and a sequence of propositions explaining why the triplet loss allows for identifiability of the reward process, with a clear flow between statements. - Lack of baselines: No alternative or baseline seems considered in the paper. The setting is perhaps a bit unusual, but since the data considered is essentially sequence of observations, any model of sequences could be used to produce embeddings (for instance, recurrent VAE, autoregressive models, time-contrastive methods) and compare the different trajectory representations. - Unclear generalization to more complex problems: (6) only implies (7) as far as I can tell, is sufficient but not necessary, and may be too strong of a condition to enforce; similary, theoretically requiring injectivity of phi seems practically too strong of a condition, as it essentially requires hashing each trajectory into a different embedding, in spite of the fact that many aspects of observations may be irrelevant for the task at hand). A key difficulty to scale this algorithm will be to find how to distance between trajectories in high dimension - perhaps the most important question for representation learning of trajectories, and a point the paper does not address. Positives: The method is overall well principled and the theoretical justification of the triplet loss is interesting. The dataset used for experiments is also interesting. Minor/Questions: - IIUC, the equation (3) needs to hold for all times T, this needs to be clearly stated as often T is used to denote total episode length. - Bottom of page 4, *The construction of the task ... phi_T=phi(mathcal T(o_T, phi_{T-1}); does this not require injectivity of phi, which is only stated afterwards? - Calling the algorithm *EM* is a bit of a stretch (there are no latent variables), it is better to call it a form of coordinate ascent. - The way the EM-like algorithm is set up, it appears the gradient with respect to theta is only computed with respect to a single time step (similar to elman networks style truncated backprop) - why not use regular backpropagation through time to train though the entire sequence? - typo on section 5.1: *Grid-world problem* - It is very hard to see on figure 3 what we are supposed to see (I don*t see these as having the same destination, perhaps highlight more clearly?)
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Trajectory representation learning for Multi-Task NMRDPs planning
Expanding Non Markovian Reward Decision Processes (NMRDP) into Markov Decision Processes (MDP) enables the use of state of the art Reinforcement Learning (RL) techniques to identify optimal policies. In this paper an approach to exploring NMRDPs and expanding them into MDPs, without the prior knowledge of the reward structure, is proposed. The non Markovianity of the reward function is disentangled under the assumption that sets of similar and dissimilar trajectory batches can be sampled. More precisely, within the same batch, measuring the similarity between any couple of trajectories is permitted, although comparing trajectories from different batches is not possible. A modified version of the triplet loss is optimised to construct a representation of the trajectories under which rewards become Markovian.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Short summary of paper: The paper investigates representation learning from trajectories in a framework which generalizes MDPs (NMRDPs), in which rewards are non-Markovian and follow a hidden process (this can be seen as a more structured, special case of a POMDP). The authors suggest learning a trajectory embedding by using a triplet loss, justified by a sufficient condition for learning an embedding which corresponds to the true task id. Overall I felt the paper fell a bit short of the standards for ICLR, based on subpar writing, lack of comparison to convincing baselines, and unclear applicability to more complex environments. Main issues: - The methodological sections of the paper seems conceptually accessible while being written in an overly mathematical, meandering fashion, using heavy, sometimes unclear notation, or referring to notation from other papers. As an example, mathcal{M}^* is first introduced by reference to another paper (the notation does not really need to be introduced until the mapping is made precise later in the paper); phi(	heta)_i^j(-1) is an unnecessarily heavy, unpleasant notation for the concept used. It was unclear if T is meant to represent the total trajectory length or a time index of a given trajectory (the notation would imply the former, the way it used, the latter). Similarly equation (6) is a very mathematical way to denote a relatively simple condition (which could be written more simply by comparing inf and sup distances between two trajectory embeddings corresponding to different or identical latent h). Again, equation (7) seems like a contrived way to write a relatively simple concept, and the simple proof that (6) implies (7) is not included. I feel this section could be improved by using precise but as simple notation as possible, and a sequence of propositions explaining why the triplet loss allows for identifiability of the reward process, with a clear flow between statements. - Lack of baselines: No alternative or baseline seems considered in the paper. The setting is perhaps a bit unusual, but since the data considered is essentially sequence of observations, any model of sequences could be used to produce embeddings (for instance, recurrent VAE, autoregressive models, time-contrastive methods) and compare the different trajectory representations. - Unclear generalization to more complex problems: (6) only implies (7) as far as I can tell, is sufficient but not necessary, and may be too strong of a condition to enforce; similary, theoretically requiring injectivity of phi seems practically too strong of a condition, as it essentially requires hashing each trajectory into a different embedding, in spite of the fact that many aspects of observations may be irrelevant for the task at hand). A key difficulty to scale this algorithm will be to find how to distance between trajectories in high dimension - perhaps the most important question for representation learning of trajectories, and a point the paper does not address. Positives: The method is overall well principled and the theoretical justification of the triplet loss is interesting. The dataset used for experiments is also interesting. Minor/Questions: - IIUC, the equation (3) needs to hold for all times T, this needs to be clearly stated as often T is used to denote total episode length. - Bottom of page 4, *The construction of the task ... phi_T=phi(mathcal T(o_T, phi_{T-1}); does this not require injectivity of phi, which is only stated afterwards? - Calling the algorithm *EM* is a bit of a stretch (there are no latent variables), it is better to call it a form of coordinate ascent. - The way the EM-like algorithm is set up, it appears the gradient with respect to theta is only computed with respect to a single time step (similar to elman networks style truncated backprop) - why not use regular backpropagation through time to train though the entire sequence? - typo on section 5.1: *Grid-world problem* - It is very hard to see on figure 3 what we are supposed to see (I don*t see these as having the same destination, perhaps highlight more clearly?)
assistant
Short summary of paper: The paper investigates representation learning from trajectories in a framework which generalizes MDPs (NMRDPs), in which rewards are non-Markovian and follow a hidden process (this can be seen as a more structured, special case of a POMDP). The authors suggest learning a trajectory embedding by using a triplet loss, justified by a sufficient condition for learning an embedding which corresponds to the true task id. Overall I felt the paper fell a bit short of the standards for ICLR, based on subpar writing, lack of comparison to convincing baselines, and unclear applicability to more complex environments. Main issues: - The methodological sections of the paper seems conceptually accessible while being written in an overly mathematical, meandering fashion, using heavy, sometimes unclear notation, or referring to notation from other papers. As an example, mathcal{M}^* is first introduced by reference to another paper (the notation does not really need to be introduced until the mapping is made precise later in the paper); phi(	heta)_i^j(-1) is an unnecessarily heavy, unpleasant notation for the concept used. It was unclear if T is meant to represent the total trajectory length or a time index of a given trajectory (the notation would imply the former, the way it used, the latter). Similarly equation (6) is a very mathematical way to denote a relatively simple condition (which could be written more simply by comparing inf and sup distances between two trajectory embeddings corresponding to different or identical latent h). Again, equation (7) seems like a contrived way to write a relatively simple concept, and the simple proof that (6) implies (7) is not included. I feel this section could be improved by using precise but as simple notation as possible, and a sequence of propositions explaining why the triplet loss allows for identifiability of the reward process, with a clear flow between statements. - Lack of baselines: No alternative or baseline seems considered in the paper. The setting is perhaps a bit unusual, but since the data considered is essentially sequence of observations, any model of sequences could be used to produce embeddings (for instance, recurrent VAE, autoregressive models, time-contrastive methods) and compare the different trajectory representations. - Unclear generalization to more complex problems: (6) only implies (7) as far as I can tell, is sufficient but not necessary, and may be too strong of a condition to enforce; similary, theoretically requiring injectivity of phi seems practically too strong of a condition, as it essentially requires hashing each trajectory into a different embedding, in spite of the fact that many aspects of observations may be irrelevant for the task at hand). A key difficulty to scale this algorithm will be to find how to distance between trajectories in high dimension - perhaps the most important question for representation learning of trajectories, and a point the paper does not address. Positives: The method is overall well principled and the theoretical justification of the triplet loss is interesting. The dataset used for experiments is also interesting. Minor/Questions: - IIUC, the equation (3) needs to hold for all times T, this needs to be clearly stated as often T is used to denote total episode length. - Bottom of page 4, *The construction of the task ... phi_T=phi(mathcal T(o_T, phi_{T-1}); does this not require injectivity of phi, which is only stated afterwards? - Calling the algorithm *EM* is a bit of a stretch (there are no latent variables), it is better to call it a form of coordinate ascent. - The way the EM-like algorithm is set up, it appears the gradient with respect to theta is only computed with respect to a single time step (similar to elman networks style truncated backprop) - why not use regular backpropagation through time to train though the entire sequence? - typo on section 5.1: *Grid-world problem* - It is very hard to see on figure 3 what we are supposed to see (I don*t see these as having the same destination, perhaps highlight more clearly?)