Its weird! run crashes randomly! my second run took much longer to crash! :
/home/ShishoSama/anaconda3/lib/python3.6/site-packages/torchvision/transforms/transforms.py:397: UserWarning: The use of the transforms.RandomSizedCrop transform is deprecated, please use transforms.RandomResizedCrop instead.
  "please use transforms.RandomResizedCrop instead.")
/home/ShishoSama/anaconda3/lib/python3.6/site-packages/torchvision/transforms/transforms.py:156: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
  "please use transforms.Resize instead.")
 [resnet18] ::   0/100 ----- [[2018-05-03 15:10:19]] [Need: 00:00:00]
i:  0
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.9907402992248535
Epoch: [0][0/10010]	Time 8.867 (8.867)	Data 7.150 (7.150)	Loss 6.9907 (6.9907)	Prec@1 0.000 (0.000)	Prec@5 0.781 (0.781)
i:  1
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.086461544036865
i:  2
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.06037712097168
i:  3
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.140486240386963
i:  4
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.094625949859619
i:  5
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.328759670257568
i:  6
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.284300327301025
i:  7
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.137808322906494
i:  8
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.297905445098877
i:  9
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.161623477935791
i:  10
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.206675052642822
i:  11
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.358131408691406
i:  12
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.334465503692627
i:  13
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.099543571472168
i:  14
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.368229866027832
i:  15
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.365810871124268
i:  16
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.326346397399902
i:  17
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.321466445922852
i:  18
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.44693660736084
i:  19
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.245199203491211
i:  20
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.311239719390869
i:  21
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.303903102874756
i:  22
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.213065147399902
i:  23
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.367886066436768
i:  24
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.063688278198242
i:  25
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.990291595458984
i:  26
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.130172252655029
i:  27
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.1098103523254395
i:  28
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.089640140533447
i:  29
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.1106367111206055
i:  30
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.049175262451172
i:  31
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.112464904785156
i:  32
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.072038650512695
i:  33
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.1823554039001465
i:  34
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.088671684265137
i:  35
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.0520429611206055
i:  36
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.061910152435303
i:  37
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.959447383880615
i:  38
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.97224760055542
i:  39
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.0115132331848145
i:  40
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.01668643951416
i:  41
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.067199230194092
i:  42
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.904612064361572
i:  43
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.920835971832275
i:  44
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.9110636711120605
i:  45
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.008086204528809
i:  46
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.999253749847412
i:  47
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.9378981590271
i:  48
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.049530029296875
i:  49
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.937681198120117
i:  50
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.913029193878174
i:  51
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  7.0122599601745605
i:  52
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
loss.data[0]:  6.941130638122559
i:  53
output:  torch.Size([128, 1000])
output(target_var):  torch.Size([128])
loss:  torch.Size([1])
input.size():  128
/pytorch/torch/lib/THCUNN/ClassNLLCriterion.cu:101: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [8,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/pytorch/torch/lib/THC/generic/THCStorage.c line=36 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "imagenet_train.py", line 313, in <module>
    main()
  File "imagenet_train.py", line 140, in main
    train(train_loader, model, criterion, optimizer, epoch, log)
  File "imagenet_train.py", line 191, in train
    print('loss.data[0]: ', loss.data[0])
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generic/THCStorage.c:36
terminate called after throwing an instance of 'std::runtime_error'
  what():  cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generic/THCStorage.c:184
Aborted (core dumped)