Yes, if you are explicitly moving the model and data to a specific device, only this device will be used.
That will be the case as long as you don’t use e.g. nn.DataParallel.
Note that (depending on your code) PyTorch might create a CUDA context on other visible devices.
If you see this behavior and want to avoid it, you could mask the desired device via CUDA_VISIBLE_DEVICES=0 python script.py args.