Hi,
I’ve encountered a very weird problem today, and managed to narrow it down to something quite basic. The code below:
from torchvision.models.inception import inception_v3 as get_model
from torch import multiprocessing
import os

def check_model():
    print(os.getpid(), 'check_model start')
    net = get_model()
    print(os.getpid(), 'number of parameters', len(list(net.parameters())))
    net.cuda(device=0)
    print(os.getpid(), 'device', next(net.parameters()).device)
    print(os.getpid(), 'check_model end')

def main():
    print(os.getpid(), 'main start')
    check_model()

    p = multiprocessing.Process(target=check_model)
    p.start()
    p.join()

    print(os.getpid(), 'main end with exitcode', p.exitcode)

if __name__ == '__main__':
    main()

Fails on the 2nd call to check_model, which is done through a new process (multiprocessing.Process). Please see the output below:
13232 main start
13232 check_model start
13232 number of parameters 292
13232 device cuda:0
13232 check_model end
13242 check_model start
13242 number of parameters 292
Process Process-1:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "src/tmp2.py", line 10, in check_model
    net.cuda(device=0)
  File "/home/ubuntu/venvs/algo/lib/python3.6/site-packages/torch/nn/modules/module.py", line 249, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/ubuntu/venvs/algo/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ubuntu/venvs/algo/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ubuntu/venvs/algo/lib/python3.6/site-packages/torch/nn/modules/module.py", line 182, in _apply
    param.data = fn(param.data)
  File "/home/ubuntu/venvs/algo/lib/python3.6/site-packages/torch/nn/modules/module.py", line 249, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error (3): initialization error
13232 main end with exitcode 1

If I replace the call to multiprocessing.Process with a regular call to the check_model function, the problem disappear. Also, if I remove the line net.cuda(device=0) the problem disappear. However, I need these functionalities for my code…
Why is this happening? I’ve tried different builtin models (VGG, ResNet) to no avail.
I’ve also tried calling the check_model twice through a child process (multiprocessing.Process) and it worked. It seems it only fails when a parent process created the net and then a child process creates another and tries to transfer it to the GPU.
Any advice will be appreciated, thanks.
Ran