<h2 dir="auto">🚀 Feature</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class A(nn.Module):
    def forward(self, x):
        if torch.jit.is_scripting() or not torch.jit.is_tracing():
            return x + 1
        else:
            return x + 2

a = A()
script = torch.jit.script(a)"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">A</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>):
        <span class="pl-k">if</span> <span class="pl-s1">torch</span>.<span class="pl-s1">jit</span>.<span class="pl-en">is_scripting</span>() <span class="pl-c1">or</span> <span class="pl-c1">not</span> <span class="pl-s1">torch</span>.<span class="pl-s1">jit</span>.<span class="pl-en">is_tracing</span>():
            <span class="pl-k">return</span> <span class="pl-s1">x</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>
        <span class="pl-k">else</span>:
            <span class="pl-k">return</span> <span class="pl-s1">x</span> <span class="pl-c1">+</span> <span class="pl-c1">2</span>

<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-v">A</span>()
<span class="pl-s1">script</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">jit</span>.<span class="pl-en">script</span>(<span class="pl-s1">a</span>)</pre></div>
<p dir="auto">prints</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError: 
Python builtin &lt;built-in method _is_tracing of PyCapsule object at 0x7fd9ad2d8e70&gt; is currently not supported in Torchscript:
  File &quot;/torch/jit/_trace.py&quot;, line 979
    code with ``torch.jit.trace``) and ``False`` otherwise.
    &quot;&quot;&quot;
    return torch._C._is_tracing()
           ~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
'is_tracing' is being compiled since it was called from 'A.forward'
  File &quot;a.py&quot;, line 20
    def forward(self, x): 
        if torch.jit.is_scripting() or not torch.jit.is_tracing():
           ~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE                                                       
            return x + 1
        else:   "><pre class="notranslate"><code class="notranslate">RuntimeError: 
Python builtin &lt;built-in method _is_tracing of PyCapsule object at 0x7fd9ad2d8e70&gt; is currently not supported in Torchscript:
  File "/torch/jit/_trace.py", line 979
    code with ``torch.jit.trace``) and ``False`` otherwise.
    """
    return torch._C._is_tracing()
           ~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
'is_tracing' is being compiled since it was called from 'A.forward'
  File "a.py", line 20
    def forward(self, x): 
        if torch.jit.is_scripting() or not torch.jit.is_tracing():
           ~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE                                                       
            return x + 1
        else:   
</code></pre></div>
<h2 dir="auto">Motivation</h2>
<p dir="auto">we have some code that works in eager &amp; scripting mode, but need a different codepath in tracing mode to handle dynamic shapes.</p>
<p dir="auto">Workaround: this works, at the cost of duplicating the eager/scripting codeblock:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="       if torch.jit.is_scripting():
            return x + 1
        else:
            if not torch.jit.is_tracing():
                return x + 1
            else:
                return x + 2"><pre class="notranslate">       <span class="pl-k">if</span> <span class="pl-s1">torch</span>.<span class="pl-s1">jit</span>.<span class="pl-en">is_scripting</span>():
            <span class="pl-k">return</span> <span class="pl-s1">x</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>
        <span class="pl-k">else</span>:
            <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">torch</span>.<span class="pl-s1">jit</span>.<span class="pl-en">is_tracing</span>():
                <span class="pl-k">return</span> <span class="pl-s1">x</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>
            <span class="pl-k">else</span>:
                <span class="pl-k">return</span> <span class="pl-s1">x</span> <span class="pl-c1">+</span> <span class="pl-c1">2</span></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gmagogsfm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gmagogsfm">@gmagogsfm</a></p>
<p dir="auto">The gather function gives incorrect gradients on both CPU and GPU when using repeated indices; no warnings or errors are raised, and the documentation doesn't say anything about this. I've seen this discussed a bit on slack, but couldn't find any issues for it on GitHub.</p>
<p dir="auto">Here's a small test case (using PyTorch 0.1.12_2 on Ubuntu 16.04):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
from torch.autograd import Variable

def gather_test(dtype):
  x = torch.FloatTensor([0.1])
  i = torch.LongTensor([0, 0])
  x = Variable(x.type(dtype), requires_grad=True)
  i = Variable(i.type(dtype).long())
  y = x.gather(0, i)
  dy = torch.FloatTensor([0.2, 0.3]).type(dtype)
  y.backward(dy)
  print('dtype: ', dtype)
  print('x: ', x.data.cpu().numpy())
  print('i: ', i.data.cpu().numpy())
  print('y: ', y.data.cpu().numpy())
  print('dy: ', dy.cpu().numpy())
  print('dx: ', x.grad.data.cpu().numpy())

gather_test(torch.FloatTensor)
gather_test(torch.cuda.FloatTensor)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span>.<span class="pl-s1">autograd</span> <span class="pl-k">import</span> <span class="pl-v">Variable</span>

<span class="pl-k">def</span> <span class="pl-en">gather_test</span>(<span class="pl-s1">dtype</span>):
  <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">FloatTensor</span>([<span class="pl-c1">0.1</span>])
  <span class="pl-s1">i</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>([<span class="pl-c1">0</span>, <span class="pl-c1">0</span>])
  <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-v">Variable</span>(<span class="pl-s1">x</span>.<span class="pl-en">type</span>(<span class="pl-s1">dtype</span>), <span class="pl-s1">requires_grad</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
  <span class="pl-s1">i</span> <span class="pl-c1">=</span> <span class="pl-v">Variable</span>(<span class="pl-s1">i</span>.<span class="pl-en">type</span>(<span class="pl-s1">dtype</span>).<span class="pl-en">long</span>())
  <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">x</span>.<span class="pl-en">gather</span>(<span class="pl-c1">0</span>, <span class="pl-s1">i</span>)
  <span class="pl-s1">dy</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">FloatTensor</span>([<span class="pl-c1">0.2</span>, <span class="pl-c1">0.3</span>]).<span class="pl-en">type</span>(<span class="pl-s1">dtype</span>)
  <span class="pl-s1">y</span>.<span class="pl-en">backward</span>(<span class="pl-s1">dy</span>)
  <span class="pl-en">print</span>(<span class="pl-s">'dtype: '</span>, <span class="pl-s1">dtype</span>)
  <span class="pl-en">print</span>(<span class="pl-s">'x: '</span>, <span class="pl-s1">x</span>.<span class="pl-s1">data</span>.<span class="pl-en">cpu</span>().<span class="pl-en">numpy</span>())
  <span class="pl-en">print</span>(<span class="pl-s">'i: '</span>, <span class="pl-s1">i</span>.<span class="pl-s1">data</span>.<span class="pl-en">cpu</span>().<span class="pl-en">numpy</span>())
  <span class="pl-en">print</span>(<span class="pl-s">'y: '</span>, <span class="pl-s1">y</span>.<span class="pl-s1">data</span>.<span class="pl-en">cpu</span>().<span class="pl-en">numpy</span>())
  <span class="pl-en">print</span>(<span class="pl-s">'dy: '</span>, <span class="pl-s1">dy</span>.<span class="pl-en">cpu</span>().<span class="pl-en">numpy</span>())
  <span class="pl-en">print</span>(<span class="pl-s">'dx: '</span>, <span class="pl-s1">x</span>.<span class="pl-s1">grad</span>.<span class="pl-s1">data</span>.<span class="pl-en">cpu</span>().<span class="pl-en">numpy</span>())

<span class="pl-en">gather_test</span>(<span class="pl-s1">torch</span>.<span class="pl-v">FloatTensor</span>)
<span class="pl-en">gather_test</span>(<span class="pl-s1">torch</span>.<span class="pl-s1">cuda</span>.<span class="pl-v">FloatTensor</span>)</pre></div>
<p dir="auto">This gives the output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="dtype:  &lt;class 'torch.FloatTensor'&gt;
x:  [ 0.1]
i:  [0 0]
y:  [ 0.1  0.1]
dy:  [ 0.2         0.30000001]
dx:  [ 0.30000001]
dtype:  &lt;class 'torch.cuda.FloatTensor'&gt;
x:  [ 0.1]
i:  [0 0]
y:  [ 0.1  0.1]
dy:  [ 0.2         0.30000001]
dx:  [ 0.30000001]"><pre class="notranslate"><code class="notranslate">dtype:  &lt;class 'torch.FloatTensor'&gt;
x:  [ 0.1]
i:  [0 0]
y:  [ 0.1  0.1]
dy:  [ 0.2         0.30000001]
dx:  [ 0.30000001]
dtype:  &lt;class 'torch.cuda.FloatTensor'&gt;
x:  [ 0.1]
i:  [0 0]
y:  [ 0.1  0.1]
dy:  [ 0.2         0.30000001]
dx:  [ 0.30000001]
</code></pre></div>
<p dir="auto">For this example I expect dx to be [0.5] since the first element of x is duplicated to both elements of y; however instead of summing the elements of dy, the gather backward pass simply chooses one of the elements of dy corresponding to the element of x.</p>
<p dir="auto">In this particular example we could use <code class="notranslate">index_select</code> instead of <code class="notranslate">gather</code>, but there are a lot of situations where <code class="notranslate">index_select</code> cannot substitute for <code class="notranslate">gather</code>; it would thus be extremely useful if <code class="notranslate">gather</code> could support repeated indices.</p>