n4tman:

if I have multiple GPUs on the system I execute my code, the 2 approches only use 1 GPU?


Yes.



 n4tman:

Also what exactly does a CUDA context mean and is is something I should avoid?


The CUDA context stores the GPU kernels, runtime etc., and thus uses memory on the specified device.