My bad you are right! I thought for a couple of seconds that we could do the same as in tensorflow.
Basically, writing code without mentioning anything about device/cuda and still being able to run it on either a GPU or CPU by just setting CUDA_VISIBLE_DEVICES.
Thanks for your help!