<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>
<p dir="auto">All variants of <code class="notranslate">ReplicationPad</code> should accept 0-dim batch size tensors for both forward and backward autograd. Related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="614589851" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/38115" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/38115/hovercard" href="https://github.com/pytorch/pytorch/issues/38115">#38115</a></p>
<p dir="auto">Test code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def test_ReplicationPad1d_Forward():
    z = torch.ones(0,3,10)
    m = torch.nn.ReplicationPad1d(3)
    m(z)

def test_ReplicationPad1d_Backward():
    z = torch.ones(0,3,10)
    m = torch.nn.ReplicationPad1d(3)
    m(z).sum().backward()

def test_ReplicationPad2d_Forward():
    z = torch.ones(0,3,10,10)
    m = torch.nn.ReplicationPad2d(3)
    m(z)

def test_ReplicationPad2d_Backward():
    z = torch.ones(0,3,10,10)
    m = torch.nn.ReplicationPad2d(3)
    m(z).sum().backward()

def test_ReplicationPad3d_Forward():
    z = torch.ones(0,3,10,10,10)
    m = torch.nn.ReplicationPad3d(3)
    m(z)

def test_ReplicationPad3d_Backward():
    z = torch.ones(0,3,10,10,10)
    m = torch.nn.ReplicationPad3d(3)
    m(z).sum().backward()"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">test_ReplicationPad1d_Forward</span>():
    <span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">0</span>,<span class="pl-c1">3</span>,<span class="pl-c1">10</span>)
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">ReplicationPad1d</span>(<span class="pl-c1">3</span>)
    <span class="pl-en">m</span>(<span class="pl-s1">z</span>)

<span class="pl-k">def</span> <span class="pl-en">test_ReplicationPad1d_Backward</span>():
    <span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">0</span>,<span class="pl-c1">3</span>,<span class="pl-c1">10</span>)
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">ReplicationPad1d</span>(<span class="pl-c1">3</span>)
    <span class="pl-en">m</span>(<span class="pl-s1">z</span>).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()

<span class="pl-k">def</span> <span class="pl-en">test_ReplicationPad2d_Forward</span>():
    <span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">0</span>,<span class="pl-c1">3</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>)
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">ReplicationPad2d</span>(<span class="pl-c1">3</span>)
    <span class="pl-en">m</span>(<span class="pl-s1">z</span>)

<span class="pl-k">def</span> <span class="pl-en">test_ReplicationPad2d_Backward</span>():
    <span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">0</span>,<span class="pl-c1">3</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>)
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">ReplicationPad2d</span>(<span class="pl-c1">3</span>)
    <span class="pl-en">m</span>(<span class="pl-s1">z</span>).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()

<span class="pl-k">def</span> <span class="pl-en">test_ReplicationPad3d_Forward</span>():
    <span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">0</span>,<span class="pl-c1">3</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>)
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">ReplicationPad3d</span>(<span class="pl-c1">3</span>)
    <span class="pl-en">m</span>(<span class="pl-s1">z</span>)

<span class="pl-k">def</span> <span class="pl-en">test_ReplicationPad3d_Backward</span>():
    <span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">ones</span>(<span class="pl-c1">0</span>,<span class="pl-c1">3</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>)
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">ReplicationPad3d</span>(<span class="pl-c1">3</span>)
    <span class="pl-en">m</span>(<span class="pl-s1">z</span>).<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()</pre></div>
<p dir="auto">I've found out that gradient of <code class="notranslate">index_select</code> (i..e <code class="notranslate">index_add</code>) behaves non-deterministically - and wrong - on GPU for some rather random conditions. The following code illustrates the problem:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import torch
from torch.autograd import Variable

def set_seed(seed):
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)

def main():
    set_seed(1)
    cuda = True
    n, e, c = 5, 100, 5
    input = torch.randn(n,c)
    idxn = torch.from_numpy(np.random.randint(n,size=e))  # indices are repeated

    if cuda:
        input = input.cuda(); idxn = idxn.cuda()

    gradsI, gradsS = [], []
    N = 4
    for i in range(N):
        inputv = Variable(input, requires_grad=True)
        sel_input = torch.index_select(inputv, 0, Variable(idxn))
        sel_input.retain_grad()

        # the following computation is one of the &quot;random conditions&quot;
        data = [torch.sum(sel_input.narrow(0,0,e//2), 0),
                torch.sum(sel_input.narrow(0,e//2,e//2), 0) ]
        out = torch.cat(data,0)
        out.exp().sum().backward()

        gradsI.append(inputv.grad.data.cpu().clone())
        gradsS.append(sel_input.grad.data.cpu().clone())

    for i in range(N):
        for j in range(N):
            print(i,j,(gradsI[i]-gradsI[j]).abs().max(), (gradsS[i]-gradsS[j]).abs().max())"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">from</span> <span class="pl-s1">torch</span>.<span class="pl-s1">autograd</span> <span class="pl-k">import</span> <span class="pl-v">Variable</span>

<span class="pl-k">def</span> <span class="pl-en">set_seed</span>(<span class="pl-s1">seed</span>):
    <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-s1">seed</span>)
    <span class="pl-s1">torch</span>.<span class="pl-en">manual_seed</span>(<span class="pl-s1">seed</span>)
    <span class="pl-s1">torch</span>.<span class="pl-s1">cuda</span>.<span class="pl-en">manual_seed</span>(<span class="pl-s1">seed</span>)

<span class="pl-k">def</span> <span class="pl-en">main</span>():
    <span class="pl-en">set_seed</span>(<span class="pl-c1">1</span>)
    <span class="pl-s1">cuda</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
    <span class="pl-s1">n</span>, <span class="pl-s1">e</span>, <span class="pl-s1">c</span> <span class="pl-c1">=</span> <span class="pl-c1">5</span>, <span class="pl-c1">100</span>, <span class="pl-c1">5</span>
    <span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">randn</span>(<span class="pl-s1">n</span>,<span class="pl-s1">c</span>)
    <span class="pl-s1">idxn</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">from_numpy</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-s1">n</span>,<span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-s1">e</span>))  <span class="pl-c"># indices are repeated</span>

    <span class="pl-k">if</span> <span class="pl-s1">cuda</span>:
        <span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s1">input</span>.<span class="pl-en">cuda</span>(); <span class="pl-s1">idxn</span> <span class="pl-c1">=</span> <span class="pl-s1">idxn</span>.<span class="pl-en">cuda</span>()

    <span class="pl-s1">gradsI</span>, <span class="pl-s1">gradsS</span> <span class="pl-c1">=</span> [], []
    <span class="pl-v">N</span> <span class="pl-c1">=</span> <span class="pl-c1">4</span>
    <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-v">N</span>):
        <span class="pl-s1">inputv</span> <span class="pl-c1">=</span> <span class="pl-v">Variable</span>(<span class="pl-s1">input</span>, <span class="pl-s1">requires_grad</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
        <span class="pl-s1">sel_input</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">index_select</span>(<span class="pl-s1">inputv</span>, <span class="pl-c1">0</span>, <span class="pl-v">Variable</span>(<span class="pl-s1">idxn</span>))
        <span class="pl-s1">sel_input</span>.<span class="pl-en">retain_grad</span>()

        <span class="pl-c"># the following computation is one of the "random conditions"</span>
        <span class="pl-s1">data</span> <span class="pl-c1">=</span> [<span class="pl-s1">torch</span>.<span class="pl-en">sum</span>(<span class="pl-s1">sel_input</span>.<span class="pl-en">narrow</span>(<span class="pl-c1">0</span>,<span class="pl-c1">0</span>,<span class="pl-s1">e</span><span class="pl-c1">//</span><span class="pl-c1">2</span>), <span class="pl-c1">0</span>),
                <span class="pl-s1">torch</span>.<span class="pl-en">sum</span>(<span class="pl-s1">sel_input</span>.<span class="pl-en">narrow</span>(<span class="pl-c1">0</span>,<span class="pl-s1">e</span><span class="pl-c1">//</span><span class="pl-c1">2</span>,<span class="pl-s1">e</span><span class="pl-c1">//</span><span class="pl-c1">2</span>), <span class="pl-c1">0</span>) ]
        <span class="pl-s1">out</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">cat</span>(<span class="pl-s1">data</span>,<span class="pl-c1">0</span>)
        <span class="pl-s1">out</span>.<span class="pl-en">exp</span>().<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()

        <span class="pl-s1">gradsI</span>.<span class="pl-en">append</span>(<span class="pl-s1">inputv</span>.<span class="pl-s1">grad</span>.<span class="pl-s1">data</span>.<span class="pl-en">cpu</span>().<span class="pl-en">clone</span>())
        <span class="pl-s1">gradsS</span>.<span class="pl-en">append</span>(<span class="pl-s1">sel_input</span>.<span class="pl-s1">grad</span>.<span class="pl-s1">data</span>.<span class="pl-en">cpu</span>().<span class="pl-en">clone</span>())

    <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-v">N</span>):
        <span class="pl-k">for</span> <span class="pl-s1">j</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-v">N</span>):
            <span class="pl-en">print</span>(<span class="pl-s1">i</span>,<span class="pl-s1">j</span>,(<span class="pl-s1">gradsI</span>[<span class="pl-s1">i</span>]<span class="pl-c1">-</span><span class="pl-s1">gradsI</span>[<span class="pl-s1">j</span>]).<span class="pl-en">abs</span>().<span class="pl-en">max</span>(), (<span class="pl-s1">gradsS</span>[<span class="pl-s1">i</span>]<span class="pl-c1">-</span><span class="pl-s1">gradsS</span>[<span class="pl-s1">j</span>]).<span class="pl-en">abs</span>().<span class="pl-en">max</span>())</pre></div>
<p dir="auto">If <code class="notranslate">cuda=False</code>, everything works and 0s are printed, i.e. all gradients are equal. However, with cuda on many nonzero differences in <code class="notranslate">gradsI</code> but not in <code class="notranslate">gradsS</code> pop up, sometimes of high magnitude, e.g.:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="0 0 0.0 0.0
0 1 1.4411518807585587e+17 0.0
0 2 1.4411518807585587e+17 0.0
0 3 1.4411518807585587e+17 0.0
1 0 1.4411518807585587e+17 0.0
1 1 0.0 0.0
1 2 3.602879701896397e+16 0.0
1 3 0.015625 0.0
2 0 1.4411518807585587e+17 0.0
2 1 3.602879701896397e+16 0.0
2 2 0.0 0.0
2 3 3.602879701896397e+16 0.0
3 0 1.4411518807585587e+17 0.0
3 1 0.015625 0.0
3 2 3.602879701896397e+16 0.0
3 3 0.0 0.0"><pre class="notranslate"><code class="notranslate">0 0 0.0 0.0
0 1 1.4411518807585587e+17 0.0
0 2 1.4411518807585587e+17 0.0
0 3 1.4411518807585587e+17 0.0
1 0 1.4411518807585587e+17 0.0
1 1 0.0 0.0
1 2 3.602879701896397e+16 0.0
1 3 0.015625 0.0
2 0 1.4411518807585587e+17 0.0
2 1 3.602879701896397e+16 0.0
2 2 0.0 0.0
2 3 3.602879701896397e+16 0.0
3 0 1.4411518807585587e+17 0.0
3 1 0.015625 0.0
3 2 3.602879701896397e+16 0.0
3 3 0.0 0.0
</code></pre></div>
<p dir="auto">This indicates that there is something wrong with backward of <code class="notranslate">torch.index_select</code>. Setting <code class="notranslate">torch.set_default_tensor_type('torch.DoubleTensor')</code> doesn't help. Can anybody shed light onto this?</p>