Yeah I checked those already. Resetting GPU didnt made it work but downgrading pytorch with conda install pytorch=0.4.1=py36_cuda9.0.176_cudnn7.1.2_1 -c pytorch just solved the problem. I mean… 1.0.1(pytorch=1.0.1=py3.6_cuda9.0.176_cudnn7.4.2_2 -c pytorch) also worked great before I tried those things (@contextmanager thing) above.
Happy for those problems seems gone but not really understand what’s been problem how was it solved by what…
I’m using nvidia-docker on driver configuration like below does this kind of things happen usually? Im not getting it at all
#image I used
$docker images 
nvidia/cuda   9.1-base   ab8ac75abb13    6 months ago   133MB

$nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.116                Driver Version: 390.116                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:01:00.0 Off |                  N/A |
| 23%   28C    P8    16W / 250W |     22MiB / 12194MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:07:00.0 Off |                  N/A |
| 23%   23C    P8     8W / 250W |      2MiB / 12196MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

$nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

If you noticed, my nvcc version == 9.1.85 but pytorch from -c pytorch with the same version (like 4.0.0 with 9.1.85) doesnt worked at all. Things are getting ridiculous for me.