Off-Policy Actor-Critic (Off-PAC) methods have proven successful in a variety of continuous control tasks. Normally, the critic's action-value function is updated using temporal-difference, and the critic in turn provides a loss for the actor that trains it to take actions with higher expected return. In this paper, we introduce a novel and flexible meta-critic that observes the learning process and meta-learns an additional loss for the actor that accelerates and improves actor-critic learning. Compared to the vanilla critic, the meta-critic network is explicitly trained to accelerate the learning process; and compared to existing meta-learning algorithms, meta-critic is rapidly learned online for a single task, rather than slowly over a family of tasks. Crucially, our meta-critic framework is designed for off-policy based learners, which currently provide state-of-the-art reinforcement learning sample efficiency. We demonstrate that online meta-critic learning leads to improvements in a variety of continuous control environments when combined with contemporary Off-PAC methods DDPG, TD3 and the state-of-the-art SAC.
This paper proposes a meta reinforcement learning approach where a meta-critic is trained in addition to a conventional actor and critic, so that the meta-critic boosts the training of the actor over a single task. The approach is combined to DDPG, TD3 and SAC and is claimed to convey superior performance (or learning speed) over these state-of-the-art actor-critic algorithms.
At first glance, the paper looks convincing, but closer inspection reveals a potential issue that I would like the authors to discuss.
The first cue is that in Fig. 6, the meta-loss does not seem to converge to anything, as the authors say it just fluctuates. Shouldn't it converge once the actor converges to close to optimal performance? 
The second cue is that appart from the rllab tasks (ant and half-cheetah), it is not clear that the meta-critic approach brings some gain in the end of training. Particularly in Reacher (Fig 7), the performance seems to collapse faster with DDPG-MC than with DDPG, and on the rest of curves of Figs 7 and 8 it is had to determine whether the MC approach brings something significant or not. And in Fig. 3, in Walker2D, TD3-MC looks rather unstable.
The third cue is that in Section 3.2, the paragraphs "Updating MC parameters" and "Designing MC" are rather unclear (I'll come back to that) and lack a theoretical justification.
So I'm wondering what exactly MC is doing and I would like to see a more detailled analysis. Couldn't a similar performance improvement be obtained by just increasing the actor learning rate and addiing some noise to the actor learning gradient? Isn't this more or less what the meta-loss does, when looking at Fig. 6?
To me, unless the authors give a clear answer to the points above, the paper should be rejected as it does not provide clear enough evidence and justification in favor of the proposed meta-learning approach.
As stated above, I found the paragraph "Updating MC parameters" lacking a principled justification.
About "Designing MC", it could be much clearer. You first express two requirements (i) and (ii). Fine.
Then you explain how to meet requirement (i), but you say this is not what you are doing (!), and then you try to explain what you are doing but without going back to the requirements. In particular, it is not clear at all why your design is "permutation invariant". You should be much more direct. Besides, the way to extract features and the relationship to batch-wise set-embedding should be made more explicit. Well, it is complicated and should be explained more clearly...
"TD3 borrows the Double Q learning idea...": no, TD3 does more than this, as it uses the min between both critics, which double Q-learning and DDQN do not.
You use HalfCheetah-v2 from gym-mujoco and HalfCheetah from rllab. Why? One might suspect this is because the performance of the MC appraoch is only better with rllab...
You say you are using 10 million steps, but this is not always the case.
It is interesting to see that the performance of TD3 on Half-Cheetah collapses at around 4 million steps. Any idea why? It seems that in many papers experiments are stopped before performance collapses, and a strong study about this phenomenon is missing as the authors don't want to show this. This is an open call to readers: a paper on that would be great! :)
"SAC-MC" gives a clear-boost for several tasks": well, looking at the figures it is not so clear. Do you mean faster learning, higher final performance or both? I would like to see this claim backed-up by some proper statistical significance test and a clear specification of the number of seeds, etc. Fig.5 conveys the adequate data for such test if your claim is that this is the final performance that matters (but beware that curves are crossing eachother, so the final performance depends a lot on where you stop...).
typos:
p1 "For example, ... (Zheng et al., 2018)." is not a sentence (no main verb).
p2 "the on-policy (approach?) needs to interact with (the) environment..."
p2 "is less effective than off-policy transitions." => unclear statement
p3 to assist(s)
Equations (2), (6) and (8) should finish with a dot as they close a sentence.
Eq X => Eq (X) : use \eqref{label} instead of just \ref{label}
p5 It's input => Its
p5 two key technologies: I would not call this a technology. Ingredients?
p5 "computational cost is ? by using": missong word
p6: asmyptotic
p9: we removing