<h2 dir="auto">Issue description</h2>
<p dir="auto">Hello all, I just found out that to do <code class="notranslate">index_copy</code> with duplicated elements in the index tensor will result in different results on CPU and GPU.<br>
To give a little context, I am trying to perform <a href="https://www.tensorflow.org/api_docs/python/tf/segment_max" rel="nofollow"><code class="notranslate">tf.segment_max</code></a> with pytorch.<br>
The elements in a tensor are associated to segments, and I would like to get all the max value in all the segments.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/00205b018b433e1d66de2e2fdf2275d2f564db4081bb16562b4f928cdac6c1dd/68747470733a2f2f7777772e74656e736f72666c6f772e6f72672f696d616765732f5365676d656e744d61782e706e67"><img src="https://camo.githubusercontent.com/00205b018b433e1d66de2e2fdf2275d2f564db4081bb16562b4f928cdac6c1dd/68747470733a2f2f7777772e74656e736f72666c6f772e6f72672f696d616765732f5365676d656e744d61782e706e67" alt="tf.segment_max" data-canonical-src="https://www.tensorflow.org/images/SegmentMax.png" style="max-width: 100%;"></a></p>
<p dir="auto">Then I found out the api <code class="notranslate">index_copy</code> and test its behavior on cpu.<br>
I realized that it will place the last segment value associated with the last segment index into the output tensor.<br>
Therefore, I just need to sort the input tensor and make the segment Index tensor reorder as the sorted input tensor.<br>
The maximum in every segment shall be the last one to copy, and the other values in the segments will be ignored. Please refer to following function <code class="notranslate">_seg_max</code>.</p>
<p dir="auto">Running on CPU, the code just run smoothly and perfectly worked as the <code class="notranslate">tf.segment_max</code>.<br>
However, on GPU, I found out that it will not keep the last value a the final output value.<br>
(The <code class="notranslate">torch.sort</code> works correctly.)</p>
<p dir="auto">I found that if there is no duplicated index in index tensor, the <code class="notranslate">index_copy</code> works just fine also on GPU.</p>
<h3 dir="auto">TL;DR</h3>
<p dir="auto"><code class="notranslate">index_copy</code> will copy the last value if it meets duplicated indices on CPU. On GPU, it behaves differently.</p>
<h2 dir="auto">Code example</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
import torch.nn as nn
import random

def _seg_max(data, seg, n_seg):
    'Perform segment max'
    _, sorted_id = torch.sort(data)
    sorted_seg = seg[sorted_id].detach()
    sorted_data = data[sorted_id].detach()
    seg_max = torch.zeros((n_seg,), device=data.device).index_copy(0, sorted_seg, sorted_data).detach()
    return seg_max

def random_seg(n_ele, n_seg):
    'Create a random segment idx list'
    seg_list = [random.randint(0, n_seg - 1) for _ in range(n_ele - n_seg)] + list(range(n_seg))
    random.shuffle(seg_list)
    return seg_list

def run(n_ele, n_seg, device):
    print('Perform `reduce max` on {}-dim vector into {} segments on {}'.format(n_ele, n_seg, device))
    with torch.no_grad():
        for _ in range(10000):
            data = torch.FloatTensor(n_ele).to(device)
            nn.init.uniform_(data)
            segment_ids = torch.LongTensor(random_seg(n_ele, n_seg)).to(device)
            seg_max = _seg_max(data, segment_ids, n_seg)
            a = seg_max.max().item()
            b = data.max().item()
            assert b == a, &quot;{} != {}&quot;.format(a, b)
    print(' -- Passed.')


gpu = torch.device('cuda')
cpu = torch.device('cpu')

run(n_ele=40, n_seg=40, device=cpu)
run(n_ele=40, n_seg=10, device=cpu)
run(n_ele=40, n_seg=40, device=gpu)
run(n_ele=40, n_seg=10, device=gpu)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span> <span class="pl-k">as</span> <span class="pl-s1">nn</span>
<span class="pl-k">import</span> <span class="pl-s1">random</span>

<span class="pl-k">def</span> <span class="pl-en">_seg_max</span>(<span class="pl-s1">data</span>, <span class="pl-s1">seg</span>, <span class="pl-s1">n_seg</span>):
    <span class="pl-s">'Perform segment max'</span>
    <span class="pl-s1">_</span>, <span class="pl-s1">sorted_id</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">sort</span>(<span class="pl-s1">data</span>)
    <span class="pl-s1">sorted_seg</span> <span class="pl-c1">=</span> <span class="pl-s1">seg</span>[<span class="pl-s1">sorted_id</span>].<span class="pl-en">detach</span>()
    <span class="pl-s1">sorted_data</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-s1">sorted_id</span>].<span class="pl-en">detach</span>()
    <span class="pl-s1">seg_max</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">zeros</span>((<span class="pl-s1">n_seg</span>,), <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s1">data</span>.<span class="pl-s1">device</span>).<span class="pl-en">index_copy</span>(<span class="pl-c1">0</span>, <span class="pl-s1">sorted_seg</span>, <span class="pl-s1">sorted_data</span>).<span class="pl-en">detach</span>()
    <span class="pl-k">return</span> <span class="pl-s1">seg_max</span>

<span class="pl-k">def</span> <span class="pl-en">random_seg</span>(<span class="pl-s1">n_ele</span>, <span class="pl-s1">n_seg</span>):
    <span class="pl-s">'Create a random segment idx list'</span>
    <span class="pl-s1">seg_list</span> <span class="pl-c1">=</span> [<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">0</span>, <span class="pl-s1">n_seg</span> <span class="pl-c1">-</span> <span class="pl-c1">1</span>) <span class="pl-k">for</span> <span class="pl-s1">_</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-s1">n_ele</span> <span class="pl-c1">-</span> <span class="pl-s1">n_seg</span>)] <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-en">range</span>(<span class="pl-s1">n_seg</span>))
    <span class="pl-s1">random</span>.<span class="pl-en">shuffle</span>(<span class="pl-s1">seg_list</span>)
    <span class="pl-k">return</span> <span class="pl-s1">seg_list</span>

<span class="pl-k">def</span> <span class="pl-en">run</span>(<span class="pl-s1">n_ele</span>, <span class="pl-s1">n_seg</span>, <span class="pl-s1">device</span>):
    <span class="pl-en">print</span>(<span class="pl-s">'Perform `reduce max` on {}-dim vector into {} segments on {}'</span>.<span class="pl-en">format</span>(<span class="pl-s1">n_ele</span>, <span class="pl-s1">n_seg</span>, <span class="pl-s1">device</span>))
    <span class="pl-k">with</span> <span class="pl-s1">torch</span>.<span class="pl-en">no_grad</span>():
        <span class="pl-k">for</span> <span class="pl-s1">_</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">10000</span>):
            <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">FloatTensor</span>(<span class="pl-s1">n_ele</span>).<span class="pl-en">to</span>(<span class="pl-s1">device</span>)
            <span class="pl-s1">nn</span>.<span class="pl-s1">init</span>.<span class="pl-en">uniform_</span>(<span class="pl-s1">data</span>)
            <span class="pl-s1">segment_ids</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">LongTensor</span>(<span class="pl-en">random_seg</span>(<span class="pl-s1">n_ele</span>, <span class="pl-s1">n_seg</span>)).<span class="pl-en">to</span>(<span class="pl-s1">device</span>)
            <span class="pl-s1">seg_max</span> <span class="pl-c1">=</span> <span class="pl-en">_seg_max</span>(<span class="pl-s1">data</span>, <span class="pl-s1">segment_ids</span>, <span class="pl-s1">n_seg</span>)
            <span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-s1">seg_max</span>.<span class="pl-en">max</span>().<span class="pl-en">item</span>()
            <span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>.<span class="pl-en">max</span>().<span class="pl-en">item</span>()
            <span class="pl-k">assert</span> <span class="pl-s1">b</span> <span class="pl-c1">==</span> <span class="pl-s1">a</span>, <span class="pl-s">"{} != {}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">a</span>, <span class="pl-s1">b</span>)
    <span class="pl-en">print</span>(<span class="pl-s">' -- Passed.'</span>)


<span class="pl-s1">gpu</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">device</span>(<span class="pl-s">'cuda'</span>)
<span class="pl-s1">cpu</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">device</span>(<span class="pl-s">'cpu'</span>)

<span class="pl-en">run</span>(<span class="pl-s1">n_ele</span><span class="pl-c1">=</span><span class="pl-c1">40</span>, <span class="pl-s1">n_seg</span><span class="pl-c1">=</span><span class="pl-c1">40</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s1">cpu</span>)
<span class="pl-en">run</span>(<span class="pl-s1">n_ele</span><span class="pl-c1">=</span><span class="pl-c1">40</span>, <span class="pl-s1">n_seg</span><span class="pl-c1">=</span><span class="pl-c1">10</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s1">cpu</span>)
<span class="pl-en">run</span>(<span class="pl-s1">n_ele</span><span class="pl-c1">=</span><span class="pl-c1">40</span>, <span class="pl-s1">n_seg</span><span class="pl-c1">=</span><span class="pl-c1">40</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s1">gpu</span>)
<span class="pl-en">run</span>(<span class="pl-s1">n_ele</span><span class="pl-c1">=</span><span class="pl-c1">40</span>, <span class="pl-s1">n_seg</span><span class="pl-c1">=</span><span class="pl-c1">10</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s1">gpu</span>)</pre></div>
<p dir="auto">Output:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Perform `segment max` on 40-dim vector into 40 segments on cpu
 -- Passed.
Perform `segment max` on 40-dim vector into 10 segments on cpu
 -- Passed.
Perform `segment max` on 40-dim vector into 40 segments on cuda
 -- Passed.
Perform `segment max` on 40-dim vector into 10 segments on cuda
Traceback (most recent call last):
  File &quot;dbg.py&quot;, line 37, in &lt;module&gt;
    run(n_ele=40, n_seg=10, device=gpu)
  File &quot;dbg.py&quot;, line 28, in run
    assert b == a, &quot;{} != {}&quot;.format(a, b)
AssertionError: 0.7791520357131958 != 0.8709555864334106"><pre class="notranslate">Perform <span class="pl-s"><span class="pl-pds">`</span>segment max<span class="pl-pds">`</span></span> on 40-dim vector into 40 segments on cpu
 -- Passed.
Perform <span class="pl-s"><span class="pl-pds">`</span>segment max<span class="pl-pds">`</span></span> on 40-dim vector into 10 segments on cpu
 -- Passed.
Perform <span class="pl-s"><span class="pl-pds">`</span>segment max<span class="pl-pds">`</span></span> on 40-dim vector into 40 segments on cuda
 -- Passed.
Perform <span class="pl-s"><span class="pl-pds">`</span>segment max<span class="pl-pds">`</span></span> on 40-dim vector into 10 segments on cuda
Traceback (most recent call last):
  File <span class="pl-s"><span class="pl-pds">"</span>dbg.py<span class="pl-pds">"</span></span>, line 37, <span class="pl-k">in</span> <span class="pl-k">&lt;</span>module<span class="pl-k">&gt;</span>
    run(n_ele=40, n_seg=10, device=gpu)
  File <span class="pl-s"><span class="pl-pds">"</span>dbg.py<span class="pl-pds">"</span></span>, line 28, <span class="pl-k">in</span> run
    assert b == a, <span class="pl-s"><span class="pl-pds">"</span>{} != {}<span class="pl-pds">"</span></span>.format(a, b)
AssertionError: 0.7791520357131958 <span class="pl-k">!</span>= 0.8709555864334106</pre></div>
<h2 dir="auto">System Info</h2>
<p dir="auto">PyTorch version: 0.4.1<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.0.176</p>
<p dir="auto">OS: Debian GNU/Linux 8.10 (jessie)<br>
GCC version: (Debian 4.9.2-10+deb8u1) 4.9.2<br>
CMake version: version 3.0.2</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: GeForce GTX TITAN X<br>
Nvidia driver version: 387.26<br>
Versions of relevant libraries:</p>
<p dir="auto">[pip3] msgpack-numpy (0.4.1)<br>
[pip3] numpy (1.14.5)<br>
[pip3] numpydoc (0.7.0)<br>
[pip3] torch (0.4.1)<br>
[pip3] torchfile (0.1.0)<br>
[pip3] torchtext (0.2.3)<br>
[pip3] torchvision (0.2.0)<br>
[conda] cuda80                    1.0                           0    soumith<br>
[conda] pytorch                   0.4.1            py36ha74772b_0<br>
[conda] torchfile                 0.1.0                     <br>
[conda] torchtext                 0.2.3                     <br>
[conda] torchvision               0.2.0                     <br>
[conda] torchvision               0.2.1                    py36_1    pytorch</p>
<h2 dir="auto">🐛 Bug</h2>
<p dir="auto">In inherited <code class="notranslate">torch.Tensor</code> ambiguous type promotions appear (nightly release).</p>
<h2 dir="auto">To Reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class MyTensor(torch.Tensor):
    ...
ten = MyTensor(10)
assert isinstance(ten, MyTensor)
assert isinstance(ten + 1, MyTensor), &quot;I expect this to fail&quot;"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">MyTensor</span>(<span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>):
    ...
<span class="pl-s1">ten</span> <span class="pl-c1">=</span> <span class="pl-v">MyTensor</span>(<span class="pl-c1">10</span>)
<span class="pl-k">assert</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">ten</span>, <span class="pl-v">MyTensor</span>)
<span class="pl-k">assert</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">ten</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>, <span class="pl-v">MyTensor</span>), <span class="pl-s">"I expect this to fail"</span></pre></div>
<h2 dir="auto">Expected behavior</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class MyTensor(torch.Tensor):
    ...
ten = MyTensor(10)
assert isinstance(ten, MyTensor)
assert not isinstance(ten + 1, MyTensor), &quot;all good&quot;"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">MyTensor</span>(<span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>):
    ...
<span class="pl-s1">ten</span> <span class="pl-c1">=</span> <span class="pl-v">MyTensor</span>(<span class="pl-c1">10</span>)
<span class="pl-k">assert</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">ten</span>, <span class="pl-v">MyTensor</span>)
<span class="pl-k">assert</span> <span class="pl-c1">not</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">ten</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>, <span class="pl-v">MyTensor</span>), <span class="pl-s">"all good"</span></pre></div>
<p dir="auto">Or a way to properly subclass from <code class="notranslate">torch.Tensor</code> avoiding such issues</p>
<h2 dir="auto">Environment</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PyTorch version: 1.8.0.dev20201011+cpu
Is debug build: True
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 10.2.0
Clang version: 10.0.1 
CMake version: version 3.18.4

Python version: 3.6 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.0.130
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A"><pre class="notranslate"><code class="notranslate">PyTorch version: 1.8.0.dev20201011+cpu
Is debug build: True
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 10.2.0
Clang version: 10.0.1 
CMake version: version 3.18.4

Python version: 3.6 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.0.130
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
</code></pre></div>
<h2 dir="auto">Additional context</h2>
<p dir="auto">I use subclassing in <a href="https://github.com/geoopt/geoopt">geoopt</a> and maintain the project</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nairbv/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nairbv">@nairbv</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mruberry/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mruberry">@mruberry</a></p>