<h2 dir="auto">Issue description</h2>
<p dir="auto">The gradient of <code class="notranslate">torch.clamp</code> when supplied with <code class="notranslate">inf</code> values is <code class="notranslate">nan</code>, even when the <code class="notranslate">max</code> parameter is specified with a finite value. Normally one would expect the gradient to be <code class="notranslate">0</code> for all values larger than <code class="notranslate">max</code>, including for <code class="notranslate">inf</code>.</p>
<h2 dir="auto">Code example</h2>
<p dir="auto">I'm trying to implement the following piecewise function: <code class="notranslate">exp(x)</code> for <code class="notranslate">x &lt; 0</code>; <code class="notranslate">1</code> otherwise. A straightforward way to do this would be to use <code class="notranslate">torch.exp(x).clamp(max=1)</code>, but this does not work as expected:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; x = torch.FloatTensor([-10, -5, 0, 5, 10, 50, 60, 70, 80, 90, 100])
&gt;&gt;&gt; x.requires_grad = True
&gt;&gt;&gt; y = torch.exp(x)
&gt;&gt;&gt; y
tensor([4.5400e-05, 6.7379e-03, 1.0000e+00, 1.4841e+02, 2.2026e+04, 5.1847e+21,
        1.1420e+26, 2.5154e+30, 5.5406e+34,        inf,        inf],
       grad_fn=&lt;ExpBackward&gt;)
&gt;&gt;&gt; z = y.clamp(max=1)
&gt;&gt;&gt; z
tensor([4.5400e-05, 6.7379e-03, 1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00,
        1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00],
       grad_fn=&lt;ClampMaxBackward&gt;)
&gt;&gt;&gt; z.sum().backward()
&gt;&gt;&gt; x.grad
tensor([0.0000, 0.0067, 1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
           nan,    nan])"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">FloatTensor</span>([<span class="pl-c1">-</span><span class="pl-c1">10</span>, <span class="pl-c1">-</span><span class="pl-c1">5</span>, <span class="pl-c1">0</span>, <span class="pl-c1">5</span>, <span class="pl-c1">10</span>, <span class="pl-c1">50</span>, <span class="pl-c1">60</span>, <span class="pl-c1">70</span>, <span class="pl-c1">80</span>, <span class="pl-c1">90</span>, <span class="pl-c1">100</span>])
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">x</span>.<span class="pl-s1">requires_grad</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">exp</span>(<span class="pl-s1">x</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">y</span>
<span class="pl-s1">tensor</span>([<span class="pl-c1">4.5400e-05</span>, <span class="pl-c1">6.7379e-03</span>, <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.4841e+02</span>, <span class="pl-c1">2.2026e+04</span>, <span class="pl-c1">5.1847e+21</span>,
        <span class="pl-c1">1.1420e+26</span>, <span class="pl-c1">2.5154e+30</span>, <span class="pl-c1">5.5406e+34</span>,        <span class="pl-s1">inf</span>,        <span class="pl-s1">inf</span>],
       <span class="pl-s1">grad_fn</span><span class="pl-c1">=</span><span class="pl-c1">&lt;</span><span class="pl-v">ExpBackward</span><span class="pl-c1">&gt;</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">z</span> <span class="pl-c1">=</span> <span class="pl-s1">y</span>.<span class="pl-en">clamp</span>(<span class="pl-s1">max</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">z</span>
<span class="pl-en">tensor</span>([<span class="pl-c1">4.5400e-05</span>, <span class="pl-c1">6.7379e-03</span>, <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.0000e+00</span>,
        <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.0000e+00</span>, <span class="pl-c1">1.0000e+00</span>],
       <span class="pl-s1">grad_fn</span><span class="pl-c1">=</span><span class="pl-c1">&lt;</span><span class="pl-v">ClampMaxBackward</span><span class="pl-c1">&gt;</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">z</span>.<span class="pl-en">sum</span>().<span class="pl-en">backward</span>()
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">x</span>.<span class="pl-s1">grad</span>
<span class="pl-en">tensor</span>([<span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0067</span>, <span class="pl-c1">1.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>, <span class="pl-c1">0.0000</span>,
           <span class="pl-s1">nan</span>,    <span class="pl-s1">nan</span>])</pre></div>
<h2 dir="auto">System Info</h2>
<p dir="auto">PyTorch version: 0.4.1<br>
Is debug build: No<br>
CUDA used to build PyTorch: 8.0.61</p>
<p dir="auto">OS: Arch Linux<br>
GCC version: (GCC) 8.1.1 20180531<br>
CMake version: Could not collect</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.2.148<br>
GPU models and configuration: GPU 0: GeForce GT 730<br>
Nvidia driver version: 396.24<br>
cuDNN version: Probably one of the following:<br>
/usr/local/R2017a/bin/glnxa64/libcudnn.so.5.1.5</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy (1.14.2)<br>
[pip] numpydoc (0.8.0)<br>
[pip] torch (0.4.1)<br>
[pip] torchvision (0.2.1)<br>
[conda] cuda80                    1.0                           0    soumith<br>
[conda] pytorch                   0.4.1           py36_cuda8.0.61_cudnn7.1.2_1  [cuda80]  pytorch<br>
[conda] torchvision               0.2.1                    py36_1    pytorch</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pearu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pearu">@pearu</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nikitaved/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nikitaved">@nikitaved</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/soulitzer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/soulitzer">@soulitzer</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>
<p dir="auto">Recurrent Dropout for RNN, GRU and LSTM.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">The concept of dropout to sequence models has been proposed in this <a href="https://arxiv.org/pdf/1512.05287.pdf" rel="nofollow">paper</a>. Also popular libraries like Keras and Tensorflow have their native implemenation of this but Pytorch does not. I personally feel Pytorch is giving a very stiff competition to Keras and Tensorflow, hence must not leave out this feature.</p>
<h2 dir="auto">Pitch</h2>
<p dir="auto">There has been some good results in the <a href="https://arxiv.org/pdf/1512.05287.pdf" rel="nofollow">paper</a> for using droupouts in sequence models for language modelling. Hence adding this to Pytorch would be of great use to Pytorch users rather than depending on alternative solution.</p>

<h2 dir="auto">Alternatives</h2>
<p dir="auto">An alternate solution is to depend on a non-native implementation of the same is given <a href="https://github.com/salesforce/awd-lstm-lm/">here</a> as part of the language modelling toolkit of Salesforce.</p>