title: In-place operations with PyTorch
body: <p>I was wondering how to deal with in-place operations in PyTorch. As I remember using in-place operation with autograd has always been problematic.</p>
  
  <p>And actually I’m surprised that this code below works, even though I haven’t tested it I believe this code would have raised an error in version <code>0.3.1</code>.</p>
  
  <p>Basically I want do is set a certain position of a tensor vector to a certain value in a like:</p>
  
  <pre><code>my_tensor[i] = 42
  </code></pre>
  
  <p><strong>Working example code:</strong></p>
  
  <pre><code># test parameter a
  a = torch.rand((2), requires_grad=True)
  print('a ', a)
  b = torch.rand(2)
  
  # calculation
  c = a + b
  
  # performing in-place operation
  c[0] = 0
  print('c ', c)
  s = torch.sum(c)
  print('s ', s)
  
  # calling backward()
  s.backward()
  
  # optimizer step
  optim = torch.optim.Adam(params=[a], lr=0.5)
  optim.step()
  
  # changed parameter a
  print('changed a', a)
  </code></pre>
  
  <p><strong>Output:</strong></p>
  
  <pre><code>a  tensor([0.2441, 0.2589], requires_grad=True)
  c  tensor([0.0000, 1.1511], grad_fn=&lt;CopySlices&gt;)
  s  tensor(1.1511, grad_fn=&lt;SumBackward0&gt;)
  changed a tensor([ 0.2441, -0.2411], requires_grad=True)
  </code></pre>
  
  <p>So obviously in version <code>0.4.1</code>. this works just fine without warnings or errors.</p>
  
  <p><em>Referring to this article in the documentation: <a href="https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd" rel="noreferrer">autograd-mechanics</a></em></p>
  
  <blockquote>
    <p>Supporting in-place operations in autograd is a hard matter, and <strong>we
    discourage their use in most cases</strong>. Autograd’s aggressive buffer
    freeing and reuse makes it very efficient and there are very few
    occasions when in-place operations actually lower memory usage by any
    significant amount. <strong>Unless you’re operating under heavy memory
    pressure, you might never need to use them.</strong></p>
  </blockquote>
  
  <p>But even though it works, <em>the use of in-place operations is discouraged in most cases.</em></p>
  
  <hr>
  
  <p>So my questions are:</p>
  
  <ul>
  <li><p><strong>How <em>much</em> does the usage of in-place operations affect performance?</strong></p></li>
  <li><p><strong>How do I get around using in-place operations in such cases where I want to set one element of a tensor to a certain value?</strong></p></li>
  </ul>
  
  <p>Thanks in advance!</p>