Off-policy temporal difference (TD) methods are a powerful class of reinforcement learning (RL) algorithms. Intriguingly, deep off-policy TD algorithms are not commonly used in combination with feature normalization techniques, despite positive effects of normalization in other domains. We show that naive application of existing normalization techniques is indeed not effective, but that well-designed normalization improves optimization stability and removes the necessity of target networks. In particular, we introduce a normalization based on a mixture of on-and off-policy transitions, which we call cross-normalization. It can be regarded as an extension of batch normalization that re-centers data for two different distributions, as present in off-policy learning. Applied to DDPG and TD3, cross-normalization improves over the state of the art across a range of MuJoCo benchmark tasks.
This paper studies the problem of feature normalization in off-policy RL, more specifically, learning a Q function with continuous action from off-policy data. It shows standard feature normalization methods in supervised learning is indeed not effective for RL settings, due to the fact that a and a’ are from very different distributions with different dynamics. Since the batch of a and a’ come to model iteratively, standard normalization method suffers from this 2-periodic distribution shift. This paper proposes a normalization method, by merging a and a’ into a * single* update step to the batch normalization layer.
This paper does catch a problem and uses a straightforward solution but empirically effective, but I still have two main concerns. 1) This paper only shows benefit 4 tasks in the MoJoCo domain. 2) As the solution is relatively straightforward, with very restrictive applicable settings (particular normalization trick with particular function approximator). It’s less clear to me whether this provides enough contribution and inspiration to other work as a conference paper. I tend to vote for reject at this time.
I would like to first point out the pros of this paper from my perspective then explain my main concerns point by point. This paper does a great job of capturing the dilemma of batch normalization in RL settings. My understanding is that the problem is caused by a periodic distribution shift between a and a’. Because we have to pass a and a’ in separate batch and BatchNorm does an online updating after each batch, we are in a dilemma, as the paper pointed out. If we don’t update BatchNorm in one of them (e.g. target value) that will be biased and make BatchNorm ineffective, and if we do so we will have a systematic difference between Q(s, a) and Q(s’, a’).
Main concerns:
1) This paper only shows benefit 4 tasks in the MoJoCo domain. Given that the empirical result is pretty much the only support of the claim in this paper, the lack of more diverse experiments would weaken the contribution.
2) The dilemma is totally caused by that BatchNorm will immediately perform update according to a batch after it is input, then a lazy update will cancel this: do a single update to the BatchNorm layers, for two batches of data (a, a’). This is equivalent to the proposed solution when alpha=0.5. It needs not to be a weakness for the algorithm itself as we appreciate simple but effective algorithm. However this makes the problem itself more like a design weakness of BatchNorm and a simple patch to fix it. I doubt how much algorithmic insight this paper could contribute, to inspire related research.
Minor point: It also makes me doubt whether we really need an alpha or not.
3) As the paper pointed out “using the training mode in the target calculation would result in different mean subtractions of the Q function and its target.” This means it will have a systematic difference in Q function used to compute Q(s, a) and Q(s’, a’), but isn’t this also true for the target network since we are using a different network to compute Q(s’, a’). Eventually, if the policy converges, those differences will disappear. So why target network will have no problem but this will. In general, I’d like to see a more clear analysis about the dilemma of BatchNorm in off-policy data, and why the two simple ways won’t work.