This shouldn’t happen and each process should use one GPU and thus create one CUDA context.
Are you calling CUDA operations on all devices in the script or did you write device-agnostic code, which only uses a single GPU?