Ok, I see.
Could you try to use
CUDA_VISIBLE_DEVICES=gpu_idx python main.py

and see if this problem still occurs.
Also, since you are using 0.3.x, I would suggest to update to the latest stable release 0.4.0.
It has some nice features and bug fixes. Have a look at the Migration Guide 3.
Unfortunately, I cannot test your script on my machine at the moment, since it’s busy.