Reinforcement learning methods have recently achieved impressive results on a wide range of control problems. However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution. This limitation largely prohibits their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control. Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account rewardbased control policies and is not capable to provide independent control policies. In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments. We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality. Additionally, we demonstrate the applicability of the method to MuJoCo control problems.
This paper proposes leveraging labelled controlled data to accelerate reinforcement-based learning of a control policy.  It provides two main contributions: pre-training the policy network of a DDPG agent in a supervised manner so that it begins in reasonable state-action distribution and regalurizing the Q-updates of the q-network to be biased towards existing actions.  The authors use the TORCS enviroment to demonstrate the performance of their method both in final cumulative return of the policy and speed of learning.
This paper is easy to understand but has a couple shortcomings and some fatal (but reparable) flaws:.
1) When using RL please try to standardize your notation to that used by the community, it makes things much easier to read.  I would strongly suggest avoiding your notation a(x|\Theta) and using \pi(x) (subscripting theta or making conditional is somewhat less important).  Your a(.) function seems to be the policy here, which is invariable denoted \pi in the RL literature.  There has been recent effort to clean up RL notation which is presented here: https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf. You have no obligation to use this notation but it does make reading of your paper much easier on others in the community.  This is more of a shortcoming than a fundamental issue.
2) More fatally, you have failed to compare your algorithm's performance against benchline implementations of similar algorithms.  It is almost trivial to run DDPG on Torcs using the openAI baselines package [https://github.com/openai/baselines].  I would have loved, for example, to see the effects of simply pre-training the DDPG actor on supervised data, vs. adding your mixture loss on the critic.  Using the baselines would have (maybe) made a very compelling graph showing DDPG, DDPG + actor pre-training, and then your complete method.
3) And finally, perhaps complementary to point 2), you really need to provide examples on more than one environment.  Each of these simulated environments has its own pathologies linked to determenism, reward structure, and other environment particularities.  Almost every algorithm I've seen published will often beat baselines on one environment and then fail to improve or even be wors on others, so it is important to at least run on a series of these.  Mujoco + AI Gym should make this really easy to do (for reference, I have no relatinship with OpenAI).  Running at least cartpole (which is a very well understood control task), and then perhaps reacher, swimmer, half-cheetah etc. using a known contoller as your behavior policy (behavior policy is a good term for your data-generating policy.)
4) In terms of state of the art you are very close to Todd Hester et. al's paper on imitation learning, and although you cite it, you should contrast your approach more clearly with the one in that paper.  Please also have a look at some more recent work my Matej Vecerik, Todd Hester & Jon Scholz: 'Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards' for an approach that is pretty similar to yours.
Overall I think your intuitions and ideas are good, but the paper does not do a good enough job justifying empirically that your approach provides any advantages over existing methods.  The idea of pre-training the policy net has been tried before (although I can't find a published reference) and in my experience will help on certain problems, and hinder on others, primarily because the policy network is already 'overfit' somewhat to the expert, and may have a hard time moving to a more optimal space.  Because of this experience I would need more supporting evidence that your method actually generalizes to more than one RL environment.