title: PyTorch out of GPU memory just after initialization
body: <p>I built my model in PyTorch. <strong>The code works well on CPU</strong>. However, it seems to be running out of GPU memory <strong>just after initializing the network and switching it to cuda</strong>. </p>
  
  <p>The failed code is:</p>
  
  <pre><code>model = init_from_scratch(args, train_exs, dev_exs)
  model.init_optimizer()
  
  nParams= sum([np.prod(list(p.size())) for p in model.network.parameters()])
  print('* total number of parameters:',nParams)
  
  device = torch.device("cuda:"+str(args.gpu) if args.cuda else "cpu")
  model.set_device(device) #call last!!
  
  # DATA ITERATORS  (not run to here)
  train_dataset = data.ReaderDataset()
  train_sampler = torch.utils.data.sampler.RandomSampler()
  train_loader = torch.utils.data.DataLoader()
  ...
  
  
  # TRAIN (not run to here)
  for epoch in range(start_epoch, args.num_epochs):
  ....
  
  def set_device(self,device):
      print("device",device)
      self.use_cuda = False if str(device) == "cpu" else True
      self.network = self.network.to(device)
  </code></pre>
  
  <ul>
  <li><p>total number of parameters: 30708906</p></li>
  <li><p>Torch version is '0.4.0' </p></li>
  <li><p>TITAN Xp with 12G GPU memory and nvidia-smi shows it free</p></li>
  </ul>
  
  <p>So, what's wrong with my code? </p>
  
  <p>The error output is:</p>
  
  <pre><code>Traceback (most recent call last):
  File "script/train2.0.py", line 683, in &lt;module&gt;
    main(args)
  File "script/train2.0.py", line 542, in main
    model.set_device(device)
  File "./model.py", line 588, in set_device
    self.network = self.network.to(device)
  File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 393, in to
    return self._apply(lambda t: t.to(device))
  File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 182, in _apply
    param.data = fn(param.data)
  File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 393, in &lt;lambda&gt;
    return self._apply(lambda t: t.to(device))
  File "/home/username/.local/lib/python3.5/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init
    torch._C._cuda_init()
  </code></pre>
  
  <p>RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCTensorRandom.cu:25</p>