Also, I found this little code to print out the currently gpu usage, which shows the first 2 gpu is not used:
In [1]: import nvidia_smi
   ...: 
   ...: nvidia_smi.nvmlInit()
   ...: 
   ...: deviceCount = nvidia_smi.nvmlDeviceGetCount()
   ...: for i in range(deviceCount):
   ...:     handle = nvidia_smi.nvmlDeviceGetHandleByIndex(i)
   ...:     info = nvidia_smi.nvmlDeviceGetMemoryInfo(handle)
   ...:     print("Device {}: {}, Memory : ({:.2f}% free): {}(total), {} (free), {} (used)".format(i, nvidia_smi.nvmlDeviceGetName(handle), 100*info.free/info.total, info.total, info.free, info.used))
   ...: 
   ...: nvidia_smi.nvmlShutdown()
Device 0: b'Tesla V100-SXM2-16GB', Memory : (100.00% free): 16914055168(total), 16913989632 (free), 65536 (used)
Device 1: b'Tesla V100-SXM2-16GB', Memory : (100.00% free): 16914055168(total), 16913989632 (free), 65536 (used)
Device 2: b'Tesla V100-SXM2-16GB', Memory : (3.82% free): 16914055168(total), 646053888 (free), 16268001280 (used)
Device 3: b'Tesla V100-SXM2-16GB', Memory : (3.82% free): 16914055168(total), 646053888 (free), 16268001280 (used)