Hi Kai @Kaixhin,
that’s cool, thanks a lot, looking forward to your standalone (cartpole) ACER implementation  (will check out your implementation in the Malmo challenge, and the Chainer code 2 in the meantime).
Michael Acar @mjacar 's discrete action TRPO works well with RMSprop, so looking forward to hopefully getting your ACER implementation and TRPO to play nicely together 
All the best,
Ajay