Thanks for sharing your code!
It seems that you keep exploring in your test.py, at line 79:
action = (mu + sigma_sq.sqrt()*Variable(eps)).data

But don’t you should just exploit with action = mu? It may explain the instability displayed by your learning curves