I found the solution to the problem that has already been solved as presented in the following links:


Order of CUDA devices 102


How to setting the GPU No. for training? 39


Gpu devices: nvidia-smi and cuda.get_device_name() output appear inconsistent 40


The solution is simply to make the first line of the following code BEFORE calling functions related to “torch.cuda” since its initialization is performed only once.
#Change the order so that it is the one used by "nvidia-smi" and not the 
#one used by all other programs ("FASTEST_FIRST")
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"

#Check that it is the same order as for "nvidia-smi":
[torch.cuda.get_device_name(i) for i in xrange(0,torch.cuda.device_count())]