Learning an off-policy predictive state representation for deep reinforcement learning for vision-based steering in autonomous driving
An algorithm is introduced for learning a predictive state representation with off-policy temporal difference (TD) learning that is then used to learn to steer a vehicle with reinforcement learning. There are three components being learned simultaneously: (1) the off-policy predictions as a compact representation of state, (2) the behavior policy distribution for estimating the off-policy predictions, and (3) the deterministic policy gradient for learning to act. A behavior policy discriminator is learned and used for estimating the important sampling ratios needed to learn the predictive representation off-policy with general value functions (GVFs). A linear deterministic policy gradient method is used to train the agent with only the predictive representations while the predictions are being learned. All three components are combined, demonstrated and evaluated on the problem of steering the vehicle from images in the TORCS racing simulator environment. Steering from only images is a challenging problem where evaluation is completed on a held-out set of tracks that were never seen during training in order to measure the generalization of the predictions and controller. Experiments show the proposed method is able to steer smoothly and navigate many but not all of the tracks available in TORCS with performance that exceeds DDPG using only images as input and approaches the performance of an ideal non-vision based kinematics model.
This paper considers the usage of learnt general value functions (GVFs) as representations for downstream control tasks in autonomous driving. The authors learn GVFs for hand-defined cumulant functions from off-policy data using TD-learning, using estimated importance weights to keep updates unbiased. A predictive representation (forecast) is chosen by the authors, which measures the “lane-centeredness” and “road angle” of the car at various effective horizons. The authors perform an experimental evaluation on TORCS using images, where they demonstrate that control policies learnt using such a predictive representation generalize better than policies which learn end-to-end directly from images. The use and discovery of predictive representations for RL, especially in the deep setting, is an interesting and important problem, relatively under-explored in the deep RL setting. The paper presents little technical novelty, however, the success of learning systems to perform control with predictive representations in this autonomous driving setting is of interest to the ICLR community. However, the paper suffers from lack of clarity and in presentation, in exposition and in the experimental section. Furthermore, I believe the experimental evaluation doesn’t shed light on the performance of the predictive representation estimation, and leaves questions open as to whether the representation improves generalization or alleviates training problems. With these concerns in hand, I believe this paper holds promise, but in its current state is not yet ready for publication at ICLR. Technical Contribution: The use of predictive state representations / forecasts is to my knowledge a relatively under-explored setting in deep RL, so the success of such representations for autonomous driving is of interest. The primary technical contribution is techniques to use a continually learnt value function as the representation for a downstream policy. The adversarial density estimation procedure introduced in the paper is not novel, and has been studied previously extensively in the community, especially in inverse RL and exploration. Presentation / Clarity: The presentation of the paper can be much improved, in detail and choice of language. Figures and their corresponding captions in Section 4 are difficult to parse. Most figures lack a legend, some lack axis labels, and captions are often non-informative. I found the exposition unclear / confusing at times, and would recommend the authors to revise the paper to improve readability in general, with focus on the introduction and experimental section. Experimental Evaluation: The experimental evaluation is missing an analysis of the learnt predictive representation (GVFs). In particular, I would like to see a stronger analysis investigating whether the learnt value functions are accurate, how fast /effectively these representations adapt to changes in the policy, how accurate the estimated density function is (using the adversarial technique), and whether this importance sampling term is necessary in practice for GVF estimation (as it seems that the experiments are run in a more online fashion). An analysis of this form would lend greater support to the argument that the proposed policy evaluation scheme is effective and useful - and that the predictive representation is actually predicting the desired quantities. No training curves are indicated anywhere, so I also have remaining questions as to whether the gain in performance presented by the authors is a function of the representation / policy generalizing better, or if the representation simplifies the RL optimization process, and thus enabling higher performance on the testing set. Furthermore, is the policy able to generalize better because of the choice of representation, or because the policy is of simplified form (linear) - would these hold for more expressive policies using this predictive representation? Finally, I would encourage the authors to include results for the comparison where the current cumulants are predicted from images as mentioned in Chen et. al 2015. The experimental setup is well explained in the main text and in the appendix. Specific Comments: 1. Citations are incorrectly formatted throughout the paper 2. In the first paragraph of Section 1.1, the bicycle model is never explained 3. The paper does not define what a predictive representation is when it is first mentioned in Section 3 4. The units for the X-axis for Figure 3 are not specified (also Figure 4, Figure 6) 5. Equation 5 is missing a constant of 2 6. As the authors deal with PSRs, the original paper introducing PSRs [1] should be cited. [1] Littman et al, *Predictive Representations of State*, NeurIPS 2001
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning an off-policy predictive state representation for deep reinforcement learning for vision-based steering in autonomous driving
An algorithm is introduced for learning a predictive state representation with off-policy temporal difference (TD) learning that is then used to learn to steer a vehicle with reinforcement learning. There are three components being learned simultaneously: (1) the off-policy predictions as a compact representation of state, (2) the behavior policy distribution for estimating the off-policy predictions, and (3) the deterministic policy gradient for learning to act. A behavior policy discriminator is learned and used for estimating the important sampling ratios needed to learn the predictive representation off-policy with general value functions (GVFs). A linear deterministic policy gradient method is used to train the agent with only the predictive representations while the predictions are being learned. All three components are combined, demonstrated and evaluated on the problem of steering the vehicle from images in the TORCS racing simulator environment. Steering from only images is a challenging problem where evaluation is completed on a held-out set of tracks that were never seen during training in order to measure the generalization of the predictions and controller. Experiments show the proposed method is able to steer smoothly and navigate many but not all of the tracks available in TORCS with performance that exceeds DDPG using only images as input and approaches the performance of an ideal non-vision based kinematics model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper considers the usage of learnt general value functions (GVFs) as representations for downstream control tasks in autonomous driving. The authors learn GVFs for hand-defined cumulant functions from off-policy data using TD-learning, using estimated importance weights to keep updates unbiased. A predictive representation (forecast) is chosen by the authors, which measures the “lane-centeredness” and “road angle” of the car at various effective horizons. The authors perform an experimental evaluation on TORCS using images, where they demonstrate that control policies learnt using such a predictive representation generalize better than policies which learn end-to-end directly from images. The use and discovery of predictive representations for RL, especially in the deep setting, is an interesting and important problem, relatively under-explored in the deep RL setting. The paper presents little technical novelty, however, the success of learning systems to perform control with predictive representations in this autonomous driving setting is of interest to the ICLR community. However, the paper suffers from lack of clarity and in presentation, in exposition and in the experimental section. Furthermore, I believe the experimental evaluation doesn’t shed light on the performance of the predictive representation estimation, and leaves questions open as to whether the representation improves generalization or alleviates training problems. With these concerns in hand, I believe this paper holds promise, but in its current state is not yet ready for publication at ICLR. Technical Contribution: The use of predictive state representations / forecasts is to my knowledge a relatively under-explored setting in deep RL, so the success of such representations for autonomous driving is of interest. The primary technical contribution is techniques to use a continually learnt value function as the representation for a downstream policy. The adversarial density estimation procedure introduced in the paper is not novel, and has been studied previously extensively in the community, especially in inverse RL and exploration. Presentation / Clarity: The presentation of the paper can be much improved, in detail and choice of language. Figures and their corresponding captions in Section 4 are difficult to parse. Most figures lack a legend, some lack axis labels, and captions are often non-informative. I found the exposition unclear / confusing at times, and would recommend the authors to revise the paper to improve readability in general, with focus on the introduction and experimental section. Experimental Evaluation: The experimental evaluation is missing an analysis of the learnt predictive representation (GVFs). In particular, I would like to see a stronger analysis investigating whether the learnt value functions are accurate, how fast /effectively these representations adapt to changes in the policy, how accurate the estimated density function is (using the adversarial technique), and whether this importance sampling term is necessary in practice for GVF estimation (as it seems that the experiments are run in a more online fashion). An analysis of this form would lend greater support to the argument that the proposed policy evaluation scheme is effective and useful - and that the predictive representation is actually predicting the desired quantities. No training curves are indicated anywhere, so I also have remaining questions as to whether the gain in performance presented by the authors is a function of the representation / policy generalizing better, or if the representation simplifies the RL optimization process, and thus enabling higher performance on the testing set. Furthermore, is the policy able to generalize better because of the choice of representation, or because the policy is of simplified form (linear) - would these hold for more expressive policies using this predictive representation? Finally, I would encourage the authors to include results for the comparison where the current cumulants are predicted from images as mentioned in Chen et. al 2015. The experimental setup is well explained in the main text and in the appendix. Specific Comments: 1. Citations are incorrectly formatted throughout the paper 2. In the first paragraph of Section 1.1, the bicycle model is never explained 3. The paper does not define what a predictive representation is when it is first mentioned in Section 3 4. The units for the X-axis for Figure 3 are not specified (also Figure 4, Figure 6) 5. Equation 5 is missing a constant of 2 6. As the authors deal with PSRs, the original paper introducing PSRs [1] should be cited. [1] Littman et al, *Predictive Representations of State*, NeurIPS 2001
assistant
This paper considers the usage of learnt general value functions (GVFs) as representations for downstream control tasks in autonomous driving. The authors learn GVFs for hand-defined cumulant functions from off-policy data using TD-learning, using estimated importance weights to keep updates unbiased. A predictive representation (forecast) is chosen by the authors, which measures the “lane-centeredness” and “road angle” of the car at various effective horizons. The authors perform an experimental evaluation on TORCS using images, where they demonstrate that control policies learnt using such a predictive representation generalize better than policies which learn end-to-end directly from images. The use and discovery of predictive representations for RL, especially in the deep setting, is an interesting and important problem, relatively under-explored in the deep RL setting. The paper presents little technical novelty, however, the success of learning systems to perform control with predictive representations in this autonomous driving setting is of interest to the ICLR community. However, the paper suffers from lack of clarity and in presentation, in exposition and in the experimental section. Furthermore, I believe the experimental evaluation doesn’t shed light on the performance of the predictive representation estimation, and leaves questions open as to whether the representation improves generalization or alleviates training problems. With these concerns in hand, I believe this paper holds promise, but in its current state is not yet ready for publication at ICLR. Technical Contribution: The use of predictive state representations / forecasts is to my knowledge a relatively under-explored setting in deep RL, so the success of such representations for autonomous driving is of interest. The primary technical contribution is techniques to use a continually learnt value function as the representation for a downstream policy. The adversarial density estimation procedure introduced in the paper is not novel, and has been studied previously extensively in the community, especially in inverse RL and exploration. Presentation / Clarity: The presentation of the paper can be much improved, in detail and choice of language. Figures and their corresponding captions in Section 4 are difficult to parse. Most figures lack a legend, some lack axis labels, and captions are often non-informative. I found the exposition unclear / confusing at times, and would recommend the authors to revise the paper to improve readability in general, with focus on the introduction and experimental section. Experimental Evaluation: The experimental evaluation is missing an analysis of the learnt predictive representation (GVFs). In particular, I would like to see a stronger analysis investigating whether the learnt value functions are accurate, how fast /effectively these representations adapt to changes in the policy, how accurate the estimated density function is (using the adversarial technique), and whether this importance sampling term is necessary in practice for GVF estimation (as it seems that the experiments are run in a more online fashion). An analysis of this form would lend greater support to the argument that the proposed policy evaluation scheme is effective and useful - and that the predictive representation is actually predicting the desired quantities. No training curves are indicated anywhere, so I also have remaining questions as to whether the gain in performance presented by the authors is a function of the representation / policy generalizing better, or if the representation simplifies the RL optimization process, and thus enabling higher performance on the testing set. Furthermore, is the policy able to generalize better because of the choice of representation, or because the policy is of simplified form (linear) - would these hold for more expressive policies using this predictive representation? Finally, I would encourage the authors to include results for the comparison where the current cumulants are predicted from images as mentioned in Chen et. al 2015. The experimental setup is well explained in the main text and in the appendix. Specific Comments: 1. Citations are incorrectly formatted throughout the paper 2. In the first paragraph of Section 1.1, the bicycle model is never explained 3. The paper does not define what a predictive representation is when it is first mentioned in Section 3 4. The units for the X-axis for Figure 3 are not specified (also Figure 4, Figure 6) 5. Equation 5 is missing a constant of 2 6. As the authors deal with PSRs, the original paper introducing PSRs [1] should be cited. [1] Littman et al, *Predictive Representations of State*, NeurIPS 2001