Sorry for the delay. I couldn’t test it on the main system so far, but I did test it on my personal system. it ran just fine!(The main script fails on this system as well).
On my personal system, I have torch 0.4.1. a TitanXp and a GTX1060 .
Here is the log :
(pytorch-py3) shisho@shisho:~/Myprojects/systempr$ nvidia-smi
Wed Dec 26 12:12:45 2018      
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   51C    P0    23W /  N/A |    840MiB /  6078MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:0C:00.0 Off |                  N/A |
| 23%   30C    P0    54W / 250W |      0MiB / 12196MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+
                                                                              
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1392      G   /usr/lib/xorg/Xorg                           663MiB |
|    0      2314      G   compiz                                       174MiB |
+-----------------------------------------------------------------------------+
(pytorch-py3) shisho@shisho:~/Myprojects/systempr$ python foo.py
Sequential(
  (0): Conv2d(3, 66, kernel_size=[3, 3], stride=(2, 2), padding=(1, 1))
  (1): BatchNorm2d(66, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (2): ReLU(inplace)
  (3): Dropout2d(p=0.0)
  (4): Conv2d(66, 128, kernel_size=[3, 3], stride=(2, 2), padding=(1, 1))
  (5): BatchNorm2d(128, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (6): ReLU(inplace)
  (7): Dropout2d(p=0.0)
  (8): Conv2d(128, 128, kernel_size=[3, 3], stride=(2, 2), padding=(1, 1))
  (9): BatchNorm2d(128, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (10): ReLU(inplace)
  (11): Dropout2d(p=0.0)
  (12): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (13): BatchNorm2d(128, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (14): ReLU(inplace)
  (15): Dropout2d(p=0.0)
  (16): Conv2d(128, 192, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (17): BatchNorm2d(192, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (18): ReLU(inplace)
  (19): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=(1, 1), ceil_mode=False)
  (20): Dropout2d(p=0.0)
  (21): Conv2d(192, 192, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (22): BatchNorm2d(192, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (23): ReLU(inplace)
  (24): Dropout2d(p=0.0)
  (25): Conv2d(192, 192, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (26): BatchNorm2d(192, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (27): ReLU(inplace)
  (28): Dropout2d(p=0.0)
  (29): Conv2d(192, 192, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (30): BatchNorm2d(192, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (31): ReLU(inplace)
  (32): Dropout2d(p=0.0)
  (33): Conv2d(192, 192, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (34): BatchNorm2d(192, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (35): ReLU(inplace)
  (36): Dropout2d(p=0.0)
  (37): Conv2d(192, 288, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (38): BatchNorm2d(288, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (39): ReLU(inplace)
  (40): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=(1, 1), ceil_mode=False)
  (41): Dropout2d(p=0.0)
  (42): Conv2d(288, 288, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (43): BatchNorm2d(288, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (44): ReLU(inplace)
  (45): Dropout2d(p=0.0)
  (46): Conv2d(288, 355, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (47): BatchNorm2d(355, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (48): ReLU(inplace)
  (49): Dropout2d(p=0.0)
  (50): Conv2d(355, 432, kernel_size=[3, 3], stride=(1, 1), padding=(1, 1))
  (51): BatchNorm2d(432, eps=1e-05, momentum=0.05, affine=True, track_running_stats=True)
  (52): ReLU(inplace)
)
/home/shisho/.virtualenvs/pytorch-py3/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py:24: UserWarning:
    There is an imbalance between your GPUs. You may want to exclude GPU 1 which
    has less than 75% of the memory or cores of GPU 0. You can do so by setting
    the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES
    environment variable.
  warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))
(pytorch-py3) shisho@shisho:~/Myprojects/systempr$

And here is the system info in case its needed :
shisho@shisho:~$ nvidia-smi
Mon Dec 24 19:55:31 2018      
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   54C    P2    23W /  N/A |   1350MiB /  6078MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:0C:00.0 Off |                  N/A |
| 24%   44C    P2    58W / 250W |    567MiB / 12196MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                              
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1630      G   /usr/lib/xorg/Xorg                           716MiB |
|    0      2634      G   compiz                                       157MiB |
|    0      3130      G   /usr/lib/firefox/firefox                       1MiB |
|    0     18774      C   python                                       471MiB |
|    1     18774      C   python                                       557MiB |
+-----------------------------------------------------------------------------+

(pytorch-py3) shisho@shisho:~/Myprojects/systempr/SimpleNetV2_Pytorch$ python
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
torch>>> torch.__version__
'0.4.1'
>>>

(pytorch-py3) shisho@shisho:~/Myprojects/systempr/SimpleNetV2_Pytorch$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148


(pytorch-py3) shisho@shisho:~/Myprojects/systempr/SimpleNetV2_Pytorch$ echo $PATH
/home/shisho/.virtualenvs/pytorch-py3/bin:/home/shisho/bin:/home/shisho/.local/bin:/usr/local/cuda-9.2/bin:/usr/local/MATLAB/R2018a/bin:/usr/local/cuda-9.2/bin:/usr/local/MATLAB/R2018a/bin:/home/shisho/bin:/home/shisho/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
(pytorch-py3) shisho@shisho:~/Myprojects/systempr/SimpleNetV2_Pytorch$ echo $LD_LIBRARY_PATH

Update :
I tried resnet18 and faced with the same exact error on the main system . so this is not a problem with the model, but the training script seems to do sth weird here!
Here is the log for resnet18 :
(pytorch3.5) shisho@o2:~/MyProjects/systempr/SimpleNetV2_Pytorch_3$ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 bash training_sequence.sh
=> creating model 'resnet18'
=> Python version : 3.5.2 (default, Nov 12 2018, 13:43:14)
=> Pytorch version : 1.0.0
=> CUDA Availability : True
=> CUDA version : 9.0.176
=> CuDNN version : 7401
=> Initial cuda seed : 5652157825403788
=> Available GPUs : 6
    => GPU(0): TITAN X (Pascal) (6,1) Total memory: 12189MB Multi processor count: 28
    => GPU(1): TITAN X (Pascal) (6,1) Total memory: 12189MB Multi processor count: 28
    => GPU(2): TITAN X (Pascal) (6,1) Total memory: 12189MB Multi processor count: 28
    => GPU(3): TITAN X (Pascal) (6,1) Total memory: 12189MB Multi processor count: 28
    => GPU(4): TITAN X (Pascal) (6,1) Total memory: 12189MB Multi processor count: 28
    => GPU(5): TITAN X (Pascal) (6,1) Total memory: 12189MB Multi processor count: 28
=> parameter : Namespace(arch='resnet18', batch_size=256, data='/home/shisho/DATASETS/tiny-imagenet-200', epochs=900, evaluate=False, lr=0.045, momentum=0.9, netidx=0, prefix='2018-12-27-5831', print_freq=200, resume='', save_dir='./snapshots/imagenet/', scale=0.0, smode=1, start_epoch=0, train_dir_name='train/', val_dir_name='val/', weight_decay=4e-05, workers=24)
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 112, 112]           9,408
            Conv2d-2         [-1, 64, 112, 112]           9,408
       BatchNorm2d-3         [-1, 64, 112, 112]             128
       BatchNorm2d-4         [-1, 64, 112, 112]             128
              ReLU-5         [-1, 64, 112, 112]               0
              ReLU-6         [-1, 64, 112, 112]               0
         MaxPool2d-7           [-1, 64, 56, 56]               0
         MaxPool2d-8           [-1, 64, 56, 56]               0
            Conv2d-9           [-1, 64, 56, 56]          36,864
           Conv2d-10           [-1, 64, 56, 56]          36,864
      BatchNorm2d-11           [-1, 64, 56, 56]             128
      BatchNorm2d-12           [-1, 64, 56, 56]             128
             ReLU-13           [-1, 64, 56, 56]               0
             ReLU-14           [-1, 64, 56, 56]               0
           Conv2d-15           [-1, 64, 56, 56]          36,864
           Conv2d-16           [-1, 64, 56, 56]          36,864
      BatchNorm2d-17           [-1, 64, 56, 56]             128
      BatchNorm2d-18           [-1, 64, 56, 56]             128
             ReLU-19           [-1, 64, 56, 56]               0
       BasicBlock-20           [-1, 64, 56, 56]               0
             ReLU-21           [-1, 64, 56, 56]               0
       BasicBlock-22           [-1, 64, 56, 56]               0
           Conv2d-23           [-1, 64, 56, 56]          36,864
           Conv2d-24           [-1, 64, 56, 56]          36,864
      BatchNorm2d-25           [-1, 64, 56, 56]             128
      BatchNorm2d-26           [-1, 64, 56, 56]             128
             ReLU-27           [-1, 64, 56, 56]               0
             ReLU-28           [-1, 64, 56, 56]               0
           Conv2d-29           [-1, 64, 56, 56]          36,864
           Conv2d-30           [-1, 64, 56, 56]          36,864
      BatchNorm2d-31           [-1, 64, 56, 56]             128
      BatchNorm2d-32           [-1, 64, 56, 56]             128
             ReLU-33           [-1, 64, 56, 56]               0
       BasicBlock-34           [-1, 64, 56, 56]               0
             ReLU-35           [-1, 64, 56, 56]               0
       BasicBlock-36           [-1, 64, 56, 56]               0
           Conv2d-37          [-1, 128, 28, 28]          73,728
           Conv2d-38          [-1, 128, 28, 28]          73,728
      BatchNorm2d-39          [-1, 128, 28, 28]             256
             ReLU-40          [-1, 128, 28, 28]               0
      BatchNorm2d-41          [-1, 128, 28, 28]             256
           Conv2d-42          [-1, 128, 28, 28]         147,456
             ReLU-43          [-1, 128, 28, 28]               0
      BatchNorm2d-44          [-1, 128, 28, 28]             256
           Conv2d-45          [-1, 128, 28, 28]         147,456
           Conv2d-46          [-1, 128, 28, 28]           8,192
      BatchNorm2d-47          [-1, 128, 28, 28]             256
      BatchNorm2d-48          [-1, 128, 28, 28]             256
           Conv2d-49          [-1, 128, 28, 28]           8,192
             ReLU-50          [-1, 128, 28, 28]               0
       BasicBlock-51          [-1, 128, 28, 28]               0
           Conv2d-52          [-1, 128, 28, 28]         147,456
      BatchNorm2d-53          [-1, 128, 28, 28]             256
      BatchNorm2d-54          [-1, 128, 28, 28]             256
             ReLU-55          [-1, 128, 28, 28]               0
             ReLU-56          [-1, 128, 28, 28]               0
       BasicBlock-57          [-1, 128, 28, 28]               0
           Conv2d-58          [-1, 128, 28, 28]         147,456
           Conv2d-59          [-1, 128, 28, 28]         147,456
      BatchNorm2d-60          [-1, 128, 28, 28]             256
      BatchNorm2d-61          [-1, 128, 28, 28]             256
             ReLU-62          [-1, 128, 28, 28]               0
             ReLU-63          [-1, 128, 28, 28]               0
       BasicBlock-64          [-1, 128, 28, 28]               0
           Conv2d-65          [-1, 128, 28, 28]         147,456
           Conv2d-66          [-1, 256, 14, 14]         294,912
      BatchNorm2d-67          [-1, 128, 28, 28]             256
      BatchNorm2d-68          [-1, 256, 14, 14]             512
             ReLU-69          [-1, 128, 28, 28]               0
       BasicBlock-70          [-1, 128, 28, 28]               0
             ReLU-71          [-1, 256, 14, 14]               0
           Conv2d-72          [-1, 256, 14, 14]         589,824
           Conv2d-73          [-1, 256, 14, 14]         294,912
      BatchNorm2d-74          [-1, 256, 14, 14]             512
      BatchNorm2d-75          [-1, 256, 14, 14]             512
           Conv2d-76          [-1, 256, 14, 14]          32,768
             ReLU-77          [-1, 256, 14, 14]               0
      BatchNorm2d-78          [-1, 256, 14, 14]             512
           Conv2d-79          [-1, 256, 14, 14]         589,824
             ReLU-80          [-1, 256, 14, 14]               0
       BasicBlock-81          [-1, 256, 14, 14]               0
      BatchNorm2d-82          [-1, 256, 14, 14]             512
           Conv2d-83          [-1, 256, 14, 14]         589,824
           Conv2d-84          [-1, 256, 14, 14]          32,768
      BatchNorm2d-85          [-1, 256, 14, 14]             512
             ReLU-86          [-1, 256, 14, 14]               0
      BatchNorm2d-87          [-1, 256, 14, 14]             512
           Conv2d-88          [-1, 256, 14, 14]         589,824
             ReLU-89          [-1, 256, 14, 14]               0
       BasicBlock-90          [-1, 256, 14, 14]               0
           Conv2d-91          [-1, 256, 14, 14]         589,824
      BatchNorm2d-92          [-1, 256, 14, 14]             512
      BatchNorm2d-93          [-1, 256, 14, 14]             512
             ReLU-94          [-1, 256, 14, 14]               0
       BasicBlock-95          [-1, 256, 14, 14]               0
             ReLU-96          [-1, 256, 14, 14]               0
           Conv2d-97            [-1, 512, 7, 7]       1,179,648
           Conv2d-98          [-1, 256, 14, 14]         589,824
      BatchNorm2d-99            [-1, 512, 7, 7]           1,024
     BatchNorm2d-100          [-1, 256, 14, 14]             512
            ReLU-101            [-1, 512, 7, 7]               0
            ReLU-102          [-1, 256, 14, 14]               0
      BasicBlock-103          [-1, 256, 14, 14]               0
          Conv2d-104            [-1, 512, 7, 7]       2,359,296
          Conv2d-105            [-1, 512, 7, 7]       1,179,648
     BatchNorm2d-106            [-1, 512, 7, 7]           1,024
     BatchNorm2d-107            [-1, 512, 7, 7]           1,024
            ReLU-108            [-1, 512, 7, 7]               0
          Conv2d-109            [-1, 512, 7, 7]         131,072
     BatchNorm2d-110            [-1, 512, 7, 7]           1,024
          Conv2d-111            [-1, 512, 7, 7]       2,359,296
            ReLU-112            [-1, 512, 7, 7]               0
      BasicBlock-113            [-1, 512, 7, 7]               0
     BatchNorm2d-114            [-1, 512, 7, 7]           1,024
          Conv2d-115            [-1, 512, 7, 7]       2,359,296
          Conv2d-116            [-1, 512, 7, 7]         131,072
     BatchNorm2d-117            [-1, 512, 7, 7]           1,024
            ReLU-118            [-1, 512, 7, 7]               0
     BatchNorm2d-119            [-1, 512, 7, 7]           1,024
          Conv2d-120            [-1, 512, 7, 7]       2,359,296
            ReLU-121            [-1, 512, 7, 7]               0
      BasicBlock-122            [-1, 512, 7, 7]               0
     BatchNorm2d-123            [-1, 512, 7, 7]           1,024
          Conv2d-124            [-1, 512, 7, 7]       2,359,296
            ReLU-125            [-1, 512, 7, 7]               0
      BasicBlock-126            [-1, 512, 7, 7]               0
       AvgPool2d-127            [-1, 512, 1, 1]               0
     BatchNorm2d-128            [-1, 512, 7, 7]           1,024
            ReLU-129            [-1, 512, 7, 7]               0
          Conv2d-130            [-1, 512, 7, 7]       2,359,296
     BatchNorm2d-131            [-1, 512, 7, 7]           1,024
            ReLU-132            [-1, 512, 7, 7]               0
      BasicBlock-133            [-1, 512, 7, 7]               0
          Linear-134                 [-1, 1000]         513,000
       AvgPool2d-135            [-1, 512, 1, 1]               0
          ResNet-136                 [-1, 1000]               0
          Linear-137                 [-1, 1000]         513,000
          ResNet-138                 [-1, 1000]               0
================================================================
Total params: 23,379,024
Trainable params: 23,379,024
Non-trainable params: 0
----------------------------------------------------------------
=> optimizer : RMSprop (
Parameter Group 0
    alpha: 0.9
    centered: False
    eps: 1
    lr: 0.045
    momentum: 0.9
    weight_decay: 4e-05
)'
FLOPs: 1816.41M, Params: 11.69M

==>>[2018-12-27 12:29:02] [Epoch=000/900] [Need: 00:00:00] [learning_rate=0.045918] [Best : Accuracy(T1/T5)=0.00/0.00, Error=100.00/100.00]
Traceback (most recent call last):
  File "imagenet_train.py", line 657, in <module>
    main()
  File "imagenet_train.py", line 288, in main
    tr_prec1, tr_prec5, tr_loss = train(train_loader, model, criterion, optimizer, epoch, log)
  File "imagenet_train.py", line 366, in train
    output = model(input_var)
  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shisho/MyProjects/systempr/SimpleNetV2_Pytorch_3/models/imagenet_resnet.py", line 125, in forward
    x = self.conv1(x)

  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shisho/.virtualenvs/pytorch3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py", line 320, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)