<h2 dir="auto">feature request description</h2>
<p dir="auto">The NLP community is shifting from LSTMs to <a href="https://arxiv.org/abs/1706.03762" rel="nofollow">Transformers</a> for a number of NLP tasks.<br>
This would be great if we could have a packed standard Transformer implementation in the 'nn' package, i.e. a <strong>nn.Transformer</strong>, just like we have a nn.LSTM.</p>
<h2 dir="auto">Code example</h2>
<p dir="auto">fairseq-py has a tested implementation of the Transformer encoder and decoder (for LM and NMT) <a href="https://github.com/pytorch/fairseq/blob/master/fairseq/models/transformer.py">https://github.com/pytorch/fairseq/blob/master/fairseq/models/transformer.py</a>. Wrapping this up in 'nn' should be the most straightforward approach to reimplementing this model.</p>
<p dir="auto">Thanks!</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">The result of <code class="notranslate">torch.max</code> with <code class="notranslate">dim</code> is different on CPU and on CUDA. CPU version return the last element, while CUDA version returns first.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [1]: import torch                                                                                                                                  

In [2]: torch.max(torch.ones(1, 4), dim=1)                                                                                                                
Out[2]: 
torch.return_types.max(
values=tensor([1.]),
indices=tensor([3]))

In [3]: torch.max(torch.ones(1, 4).cuda(), dim=1)                                                                                                         
Out[3]: 
torch.return_types.max(
values=tensor([1.], device='cuda:0'),
indices=tensor([0], device='cuda:0'))"><pre class="notranslate"><code class="notranslate">In [1]: import torch                                                                                                                                  

In [2]: torch.max(torch.ones(1, 4), dim=1)                                                                                                                
Out[2]: 
torch.return_types.max(
values=tensor([1.]),
indices=tensor([3]))

In [3]: torch.max(torch.ones(1, 4).cuda(), dim=1)                                                                                                         
Out[3]: 
torch.return_types.max(
values=tensor([1.], device='cuda:0'),
indices=tensor([0], device='cuda:0'))
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">CPU and CUDA version should give the same result for sure.</p>
<p dir="auto">In my opinion, <code class="notranslate">cuda</code> behavior is correct because it aligns with NumPy.<br>
From the other hand current <code class="notranslate">torch.argmax</code> implementation follows the cpu behaviour.</p>
<p dir="auto">upd. There is an open pull request that fixes this behavior for argmax <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="612843098" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/37864" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/37864/hovercard" href="https://github.com/pytorch/pytorch/pull/37864">#37864</a> .</p>
<h2 dir="auto">Environment</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [4]: torch.__version__                                                                                                                             
Out[4]: '1.5.0a0+8f84ded'"><pre class="notranslate"><code class="notranslate">In [4]: torch.__version__                                                                                                                             
Out[4]: '1.5.0a0+8f84ded'
</code></pre></div>
<h2 dir="auto">Additional context</h2>