The CUDA initialization will see all devices and might thus add the small amount of memory. You can hide the other devices by using CUDA_VISIBLE_DEVICES=0 python script.py args to make only certain GPUs visible inside your script.