I see a similar issue when running mytensor.to(torch.device('cuda'))
here’s the output to several calls to nvidia-smi, notice how the memory usage goes up until ~340MiB and then the call returns
Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:13 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   43C    P8    N/A /  N/A |      0MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:15 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |     33MiB /   983MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python    22MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:16 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |     94MiB /   983MiB |     15%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python    83MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:17 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    140MiB /   983MiB |      7%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   130MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:18 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    173MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   162MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:19 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    216MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   205MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:20 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    249MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   238MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:21 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    292MiB /   983MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   281MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:23 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    333MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   322MiB |
+-----------------------------------------------------------------------------+
^[[A                                                                                                                                                                                                              

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:25 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    353MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   342MiB |
+-----------------------------------------------------------------------------+