Warning never dabbled on RL yet but my gut feeling would be:
1/ Train on a simpler model, and use the pretrained weights to model more complex behaviour. (Aka fine-tuning, or even curriculum learning)
2/ Try other RL like A3C.
3/ Read Sutton’s RL book 24