Many deep reinforcement learning algorithms contain inductive biases that sculpt the agent's objective and its interface to the environment. These inductive biases can take many forms, including domain knowledge and pretuned hyperparameters. In general, there is a trade-off between generality and performance when we use such biases. Stronger biases can lead to faster learning, but weaker biases can potentially lead to more general algorithms that work on a wider class of problems. This trade-off is relevant because these inductive biases are not free; substantial effort may be required to obtain relevant domain knowledge or to tune hyper-parameters effectively. In this paper, we re-examine several domain-specific components that modify the agent's objective and environmental interface. We investigated whether the performance deteriorates when all these fixed components are replaced with adaptive solutions from the literature. In our experiments, performance sometimes decreased with the adaptive components, as one might expect when comparing to components crafted for the domain, but sometimes the adaptive components performed better. We then investigated the main benefit of having fewer domain-specific components, by comparing the learning performance of the two systems on a different set of continuous control problems, without additional tuning of either system. As hypothesized, the system with adaptive components performed better on many of the tasks.
The deep reinforcement learning (RL) community has demonstrated that well-tuned deep RL algorithms can master a wide range of challenging tasks. Human-level performance has been approached or surpassed on board-games such as Go and Chess (Silver et al., 2017), video-games such as Atari (Mnih et al., 2015;Hessel et al., 2018a;Horgan et al., 2018), and custom 3D navigation tasks (Johnson et al., 2016;Kempka et al., 2016;Beattie et al., 2016;Jaderberg et al., 2016;Espeholt et al., 2018). These results are a testament to the generality of the overall approach. At times, however, the excitement for the constant stream of new domains being mastered by suitable RL algorithms may have over-shadowed the dependency on inductive biases of these agents, and the amount of tuning that is often required for these to perform effectively in new domains. A clear example of the benefits of generality is the AlphaZero algorithm (Silver et al., 2017). AlphaZero differs from the earlier AlphaGo algorithm  by removing all dependencies on Go-specific inductive biases and human data. After removing these biases, AlphaZero could not just surpass human performance on Go but also learn to play Chess and Shogi.
In general, there is a trade off between generality and performance when we inject inductive biases into our algorithms. Inductive biases take many forms, including domain knowledge and pretuned learning parameters. If applied carefully, such biases can lead to faster and better learning. On the other hand, fewer biases can potentially lead to more general algorithms that work out of the box on a wider class of problems. Crucially, most inductive biases are not free: for instance, substantial effort can be required to attain the relevant domain knowledge or pretune parameters. This cost is often hidden-for instance, one might use hyperparameters established as good in prior work on the same domain, without knowing how much data or time was spent optimising these, nor how specific the settings are to the given domain. Systematic studies about the impact of different inductive biases are rare and the generality of these different biases is often unclear. Another consideration is that inductive biases may mask the generality of other parts of the system as a whole; if a learning algorithm tuned for a specific domain does not generalize out of the box to a new domain, it can be unclear whether it is due to the having the wrong inductive biases or whether the underpinning learning algorithm is lacking something important.
In this paper, we consider several commonly used domain heuristics, and investigate if (and by how much) performance deteriorates when we replace these with more general adaptive components, and we assess the impact of such replacements on the generality of the agent. We consider two broad ways of injecting inductive biases in RL agents: 1) sculpting the agent's objective (e.g., clipping and discounting rewards), 2) sculpting the agent-environment interface (e.g., repeating each selected action a hard-coded fixed number of times, or crafting of the learning agent's observation). Our contributions show that all the carefully crafted heuristics commonly used in Atari, and often considered essential for good performance on this domain, can be safely replaced with adaptive components, while preserving competitive performance across the benchmark. Furthermore, we show that this results in increased generality for an actor critic agent; the resulting fully adaptive system can be applied with no additional tuning on a separate suite of continuous control tasks that have different environmental interfaces. Here the adaptive system achieved much higher performance than a comparable system using the Atari tuned heuristics, and also higher performance than an actor-critic agent that was tuned for this benchmark (Tassa et al., 2018).
This paper focuses on deep reinforcement learning methods and discusses the presence of inductive biases in the existingRL algorithm. Specifically, they discuss biases that take the form of domain knowledge or hyper-parameter tuning. The authors state that such biases rise the tradeoff between generality and performance wherein strong biases can lead to efficient performance but deteriorate generalization across domains. Further, it motivates that most inductive biases has a cost associated to it and hence it is important to study and analyze the effect of such biases. 
To support their insights, the authors investigate the performance of well known actor-critic model in the Atari environment after replacing domain specific heuristics with the adaptive components. The author considers two ways of injecting biases: i) sculpting agents objective and ii) sculpting agent's environment. They show empirical evidence that replacing carefully designed heuristics to induce biases with more adaptive counterparts preserves performance and generalizes without additional fine tuning.
The paper focuses on an important concept and problem of inductive biases in deep reinforcement learning techniques. 
Analysis of such biases and methods to use them judiciously is an interesting future direction. The paper covers a lot of related work in terms of various algorithms and corresponding biases.
However, this paper only discusses such concepts at high level and provides short empirical evidences in a single environment to support their arguments. Further, both the heuristics used in practice and the adaptive counterparts that the paper uses to replace those heuristics are all available in existing approaches and there is no novel contribution in that direction too.
Finally, the adaptive methods based on parallel environment and RNNs have several limitation, as per author's own admission.
Overall, the paper does not have any novel technical contributions or theoretical analysis on the effect of such inductive biases which makes it very weak. Further, there is nothing surprising about the author's claims and many of the outcomes from the analysis are expected. The authors are recommended to consider this task more rigorously and provide stronger and concrete analysis on the effects of inductive biases on variety of algorithms and variety of environments.