<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h1>
<p dir="auto">We would like Pytorch to support the automatic mixed precision training recipe:  auto-casting of Cuda operations to FP16 or FP32 based on a whitelist-blacklist model of what precision is best for each operation, as well as gradient scaling to avoid underflow during the backward pass.</p>
<h1 dir="auto">Work in progress</h1>
<ul dir="auto">
<li><a href="https://github.com/pytorch/pytorch/pull/33366" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/33366/hovercard">PR #33366</a> for the gradient scaling API (merged!)</li>
<li><a href="https://github.com/pytorch/pytorch/pull/35102" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/35102/hovercard">PR #35102</a> for the autocasting API and eager-mode backend (merged!)</li>
<li><a href="https://github.com/pytorch/pytorch/issues/25387" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/25387/hovercard">Dedicated issue tracking jit support</a> (<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jjsjann123/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jjsjann123">@jjsjann123</a> is driving this on our side)</li>
<li><a href="https://github.com/pytorch/pytorch/issues/28762" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/28762/hovercard">Jit scripting for context managers (<code class="notranslate">with</code> statements)</a></li>
</ul>
<h1 dir="auto">Motivation</h1>
<p dir="auto">Mixed precision is essential to achieve good performance on Tensor Core GPUs (Volta + Turing).  We've been trying to educate and evangelize for it since Volta's release.  We have <a href="https://github.com/nvidia/apex">Apex</a>, our own repository of tools for mixed precision training, which has seen moderate adoption.  However, forcing users to install a separate toolkit is burdensome, especially if their environments aren't set up to build extensions, and we'll never be able to achieve the same performance as a native integration.  Native, well documented support in Pytorch core is certainly the most convenient way to enable mixed precision training for the largest number of users.</p>
<h1 dir="auto">Pitch</h1>
<p dir="auto">After initial discussions with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mruberry/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mruberry">@mruberry</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zdevito/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zdevito">@zdevito</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jamesr66a/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jamesr66a">@jamesr66a</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a>, and <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jjsjann123/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jjsjann123">@jjsjann123</a> we believe the UX should permit auto-casting and gradient scaling as modular and independent components.</p>
<h2 dir="auto">Auto-casting</h2>
<h3 dir="auto">Background</h3>
<p dir="auto">The current philosophy of Apex's Automatic Mixed Precision (Amp, in recommended mode "O1") is that when training with mixed precision, the user never needs to manually alter the precision of their model or data.  The user declares their model in default (FP32) precision.  <strong>The model parameters (leaves) are and remain FP32 for the duration of training.  These leaves are also directly owned and stepped by the optimizer,</strong> which is identical to the ordinary behavior of Pytorch in the absence of mixed precision.</p>
<p dir="auto">To ensure that FP16 is used for operations that benefit from it, <code class="notranslate">Tensor.*</code> methods and <code class="notranslate">torch.*</code> and <code class="notranslate">torch.nn.functional.*</code> functions are patched to cast data to a certain type before running the actual op.  Which type depends on what precision is best for that op.  For example, <code class="notranslate">torch.mm</code> is patched to cast the incoming input and weight to FP16, which enables Tensor Cores.  <code class="notranslate">torch.log</code> is patched to cast input to fp32, because log's forward and backward may require a large dynamic range.  This casting-as-data-flows-through-functions is the strategy used by Amp in Apex today, and achieves accuracy comparable to pure FP32 training on a wide range of networks.  It is also the strategy used by MXNet and Tensorflow's Amp integrations.  However, Apex's Amp is implemented by Python-side monkey-patching of <code class="notranslate">torch.*</code> and <code class="notranslate">torch.nn.functional.*</code> functions, which is invasive and not ideal for performance.</p>
<h3 dir="auto">Proposed Implementation</h3>
<p dir="auto">For eager execution, we propose to integrate the same casting-as-data-flows-through-functions strategy that Apex's Amp uses.  We propose to implement this by inserting the casts in some of the autogenerated C++ functions on the Cuda dispatch path.  Each such function will be given a few additional lines by the autogeneration script.  For example, a whilelist function with 1 argument would be given something like</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if(autocasting_is_enabled())
  input = input.half()"><pre class="notranslate"><span class="pl-k">if</span>(<span class="pl-en">autocasting_is_enabled</span>())
  <span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s1">input</span>.<span class="pl-en">half</span>()</pre></div>
<p dir="auto">These casts should be autograd-exposed, so they will be reversed in backward().  They should also precede the autograd-exposed call of the whitelist or blacklist op itself, so if the op saves its inputs for backward, the inputs are saved as the correct (post-cast) type.</p>
<p dir="auto">On the Python side, the user shall request auto-casting by running the forward pass (or any portions of the forward pass where auto-casting is desired) under a nestable context manager, for example</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="@contextlib.contextmanager
def autocast(whitelist_type=torch.float16, enabled=True):
    old_whitelist_type, old_status = torch.get_autocasting_state()
    torch.set_autocasting_state(whitelist_type, enabled)
    try:
        yield
    finally:
        torch.set_autocasting_state(original_whitelist_type, old_status)"><pre class="notranslate"><span class="pl-en">@<span class="pl-s1">contextlib</span>.<span class="pl-s1">contextmanager</span></span>
<span class="pl-k">def</span> <span class="pl-en">autocast</span>(<span class="pl-s1">whitelist_type</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">float16</span>, <span class="pl-s1">enabled</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    <span class="pl-s1">old_whitelist_type</span>, <span class="pl-s1">old_status</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">get_autocasting_state</span>()
    <span class="pl-s1">torch</span>.<span class="pl-en">set_autocasting_state</span>(<span class="pl-s1">whitelist_type</span>, <span class="pl-s1">enabled</span>)
    <span class="pl-k">try</span>:
        <span class="pl-k">yield</span>
    <span class="pl-k">finally</span>:
        <span class="pl-s1">torch</span>.<span class="pl-en">set_autocasting_state</span>(<span class="pl-s1">original_whitelist_type</span>, <span class="pl-s1">old_status</span>)</pre></div>
<p dir="auto"><code class="notranslate">torch.get_</code> and <code class="notranslate">set_autocasting_state</code> will get/set a backend state that is queryable within C++ dispatch functions.</p>
<p dir="auto"><code class="notranslate">whitelist_type</code> can be changed to request that whitelist functions be autocast to types other than FP16.  The <code class="notranslate">enabled</code> argument can be used to locally disable autocasting if the user wishes to have manual control over the types that are used in some regions of their model, while permitting autocasting in others.</p>
<p dir="auto">My initial thought is that <code class="notranslate">with autocast()</code> may be used to wrap any regions of code where a graph is being constructed via explicit Python invocations of Pytorch ops (ie, the forward pass), but shall not wrap any region where a previously constructed graph is being backwarded through.  All desired casting will have been recorded as part of the graph construction, and will be correctly reversed by a bare backward call without needing to be under an <code class="notranslate">autocast</code> context.  Backward passes with <code class="notranslate">create_graph=True</code> also belong to the latter category (ie, they should not be under a <code class="notranslate">with autocast()</code> context). <a href="#gradient-penalty">Gradient Penalty</a> under End to End Examples below shows this philosophy more clearly.</p>
<p dir="auto">It's possible that running both forward and backward under the context manager won't do any harm (ie, any resulting casts requested during backward will be no-ops, because the type flow is already being properly/faithfully reversed by autograd) and it can be permissible (but not required) to also allow the backward pass to take place under the context manager.</p>
<p dir="auto">When training with FP16, gradient scaling and auto-casting must both be requested by the user.  We envision that when autocasting to formats other than FP16, gradient scaling will not be necessary, and the auto-casting context manager can be used without any gradient scaling.</p>
<h3 dir="auto">Example Usage</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="with autocast():
    output = model(input)
    loss = loss_fn(output, target)
# The backward pass should be invoked outside the context manager.  All casting has been appropriately recorded as part of the forward pass."><pre class="notranslate"><span class="pl-k">with</span> <span class="pl-en">autocast</span>():
    <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
    <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
<span class="pl-c"># The backward pass should be invoked outside the context manager.  All casting has been appropriately recorded as part of the forward pass.</span></pre></div>
<p dir="auto">Within <code class="notranslate">model.forward</code>, if the user has regions where they wish explicit control over the precision, they may nest an invocation of <code class="notranslate">with autocast(enabled=False)</code>:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def forward(self, x):
    x = self.layer_permitting_autocasting(x)
    with autocast(enabled=False):
        x = x.float()
        x = self.explicitly_float_layer(x)
    x = self.another_layer_permitting_autocasting(x)
    return x"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>):
    <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">layer_permitting_autocasting</span>(<span class="pl-s1">x</span>)
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>(<span class="pl-s1">enabled</span><span class="pl-c1">=</span><span class="pl-c1">False</span>):
        <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">x</span>.<span class="pl-en">float</span>()
        <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">explicitly_float_layer</span>(<span class="pl-s1">x</span>)
    <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">another_layer_permitting_autocasting</span>(<span class="pl-s1">x</span>)
    <span class="pl-k">return</span> <span class="pl-s1">x</span></pre></div>
<h3 dir="auto">Gotchas/Challenges</h3>
<p dir="auto">The Amp casts need to be recorded by autograd, so they'll be properly reversed in backward.  Unfortunately, for many ops, dispatch into an autograd-disabled region currently occurs in <code class="notranslate">VariableType*.cpp</code>, at a higher level of the call chain than the Cuda-specific dispatch functions.  <code class="notranslate">VariableType*.cpp</code> is also the level at which necessary data is saved for backward.  In other words, by the time we've reached the Cuda-specific dispatch functions, it's too late to invoke autograd-exposed casts.  The alternative is to insert the</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if(autocasting_is_enabled())
  input = input.half() or float()"><pre class="notranslate"><span class="pl-k">if</span>(<span class="pl-en">autocasting_is_enabled</span>())
  <span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s1">input</span>.<span class="pl-en">half</span>() <span class="pl-c1">or</span> <span class="pl-en">float</span>()</pre></div>
<p dir="auto">snippets at the level of <code class="notranslate">VariableType*.cpp</code>, before we dive into autograd-disabled regions, but then these if statements will be on the hot path taken by all dispatches (Cuda and non-Cuda).  I'd like to avoid having the if statement on any non-Cuda code path.  This is a tricky problem and I need to think hard about it.</p>
<h2 dir="auto">Gradient scaling</h2>
<h3 dir="auto">Background</h3>
<p dir="auto">Late in training, FP16 gradients can underflow, halting convergence and in some cases causing destabilization.  Apex's Amp mitigates underflow via "dynamic gradient scaling."  The implementation creates scaled gradients by handing the user <code class="notranslate">scaled_loss  =  loss*scale_factor</code>, then requiring that the user invoke <code class="notranslate">scaled_loss.backward()</code>. By the chain rule, all gradients flowing backward through the network are then scaled by <code class="notranslate">scale_factor</code>.  Apex's Amp attempts to maximize use of FP16's full dynamic range by choosing the highest <code class="notranslate">scale_factor</code> that can be used without incurring inf/nan gradients, which is accomplished as follows:  Initially, a high <code class="notranslate">scale_factor</code> is chosen.  Each iteration, after backward() returns, gradients are checked for infs/nans.  If any infs/nans are found, the optimizer skips the step, and the scale factor is reduced.  The scale factor is also periodically increased if a streak of successful (inf/nan free) iterations occurs.  Gradients are unscaled in FP32 before being applied to FP32 model parameters.</p>
<h3 dir="auto">Proposed API</h3>
<p dir="auto">User scripts will implement gradient scaling with an instance of a helper class.  Typical use would look like</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = torch.cuda.amp.AmpScaler()

for input, target in data:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">cuda</span>.<span class="pl-s1">amp</span>.<span class="pl-v">AmpScaler</span>()

<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
    <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()
    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<p dir="auto">Creating scaled gradients with <code class="notranslate">torch.autograd.backward</code> would look like</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.autograd.backward(scaler.scale((output0, output1)), grad_tensors=(grad0, grad1))"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-s1">autograd</span>.<span class="pl-en">backward</span>(<span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>((<span class="pl-s1">output0</span>, <span class="pl-s1">output1</span>)), <span class="pl-s1">grad_tensors</span><span class="pl-c1">=</span>(<span class="pl-s1">grad0</span>, <span class="pl-s1">grad1</span>))</pre></div>
<p dir="auto">Creating scaled gradients with <code class="notranslate">torch.autograd.grad</code> would look like</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.autograd.grad(scaler.scale((output0, output1)), model.parameters(), grad_outputs=(grad0, grad1))"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-s1">autograd</span>.<span class="pl-en">grad</span>(<span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>((<span class="pl-s1">output0</span>, <span class="pl-s1">output1</span>)), <span class="pl-s1">model</span>.<span class="pl-en">parameters</span>(), <span class="pl-s1">grad_outputs</span><span class="pl-c1">=</span>(<span class="pl-s1">grad0</span>, <span class="pl-s1">grad1</span>))</pre></div>
<p dir="auto"><a href="https://github.com/pytorch/pytorch/blob/35829f24ef2a71eacedcd6307b2fe9325e4a6a94/torch/cuda/amp/amp_scaler.py#L6">The best explanation for <code class="notranslate">AmpScaler</code> is the class itself, as found in the gradient scaling PR</a>.  It's lightweight and each function is documented.</p>
<p dir="auto"><code class="notranslate">scaler</code> internally initializes and maintains the scale value, and updates it each iteration based on whether <code class="notranslate">optimizer</code>'s gradients contained infs or NaNs.  If infs/NaN gradients are encountered in a given iteration, <code class="notranslate">scaler.update</code> reduces the scale.  If no inf/NaN gradients are encountered, <code class="notranslate">scaler.update</code> increases the scale slightly.  This approach achieves what dynamic gradient scaling intends:  over time, riding the edge of the highest gradient scale that can be used without incurring overflow.</p>
<p dir="auto">The user also has the freedom to manually reset the scale value at the end of any iteration, by passing a new scale to <code class="notranslate">scaler.update</code> as a Tensor or Python float.</p>
<p dir="auto"><code class="notranslate">AmpScaler</code> instances contain at most a few Tensors and Python scalars.  When checkpointing, the user can easily save an <code class="notranslate">AmpScaler</code> instance as part of the checkpoint, alongside <code class="notranslate">model.state_dict()</code> and <code class="notranslate">optimizer.state_dict()</code>.  The model and optimizer instance(s) are not affected by AmpScaler, and remain exactly what they would be in the absence of mixed precision.  Therefore, behaviors and invocations of <code class="notranslate">model.state_dict()</code> and <code class="notranslate">optimizer.state_dict()</code> themselves may remain unaltered.</p>
<h3 dir="auto">Interaction with Existing Optimizers</h3>
<p dir="auto">As long as training scripts adhere to the proposed API, existing optimizers (whether native or custom) will immediately work with mixed precision.  In particular, the gradient scaling API does not rely on changes to the Python source of <code class="notranslate">step()</code> in any existing optimizers.</p>
<p dir="auto"><code class="notranslate">scaler.step(optimizer)</code> <a href="https://github.com/pytorch/pytorch/blob/35829f24ef2a71eacedcd6307b2fe9325e4a6a94/torch/cuda/amp/amp_scaler.py#L146-L192">wraps <code class="notranslate">optimizer.step()</code> with logic to make it scaling-safe</a>.  Specifically, <code class="notranslate">scaler.step(optimizer)</code></p>
<ul dir="auto">
<li>makes sure gradients are unscaled before <code class="notranslate">optimizer.step()</code></li>
<li>skips <code class="notranslate">optimizer.step()</code> if inf/nan gradients are found.</li>
</ul>
<p dir="auto"><code class="notranslate">optimizer.step()</code> itself is untouched, and again, does not need to change for existing optimizers.</p>
<h3 dir="auto">Interaction with New Custom Optimizers</h3>
<p dir="auto"><code class="notranslate">AmpScaler</code> defines a contract such that custom optimizers <em>may</em> implement their own scaling-safe <code class="notranslate">step</code> methods if they choose to.  If an optimizer obeys this contract, <a href="https://github.com/pytorch/pytorch/blob/35829f24ef2a71eacedcd6307b2fe9325e4a6a94/torch/cuda/amp/amp_scaler.py#L179-L184"><code class="notranslate">AmpScaler.step</code> will call the optimizer's <code class="notranslate">step</code> method directly</a>.  This gives custom optimizer authors a control point to implement ninja optimizations like sync-free dynamic gradient scaling.</p>
<h3 dir="auto">Gotchas/Challenges</h3>
<p dir="auto">When the user invokes a backward pass with a scale factor, all gradients produced by this backward pass (leaf gradients produced by <code class="notranslate">loss.backward</code> or <code class="notranslate">torch.autograd.backward</code>, or out-of-place gradients produced by <code class="notranslate">torch.autograd.grad</code>) will be scaled.  Therefore, anything that manipulates the gradients between <code class="notranslate">scaler.scale(loss).backward()</code> and the <code class="notranslate">scaler.step(optimizer)</code> will require proper awareness of the scale factor.  Examples of operations that require scale-factor-aware treatment are</p>
<ul dir="auto">
<li>gradient clipping</li>
<li>gradient penalty computations.</li>
</ul>
<p dir="auto">In our opinion, requiring the user be aware of the scale factor when making direct use of the gradients is not terribly burdensome.  The user has explicitly requested gradient scaling; they should not be surprised when they end up with scaled gradients on their hands.  The treatment of such cases is also not difficult from a code-change perspective, as long as it is clearly documented.</p>
<p dir="auto">Closures are also a challenge.  Do we need to support explicit closure use?  Based on the scripts and issues I've seen, closure use is not terribly common, and LBFGS is the only native optimizer that requires a closure.  However, I think I can torture the proposed API into supporting closures if it turns out to be in high demand.</p>
<h2 dir="auto">End to End Examples (Auto-Casting + Gradient Scaling)</h2>
<h3 dir="auto">Typical Use (1 loss, 1 optimizer)</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = AmpScaler()
...
for input, target in data:
    optimizer.zero_grad()
    with autocast():
        output = model(input)
        loss = loss_fn(output, target)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-v">AmpScaler</span>()
...
<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
        <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()
    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<h3 dir="auto">Gradient Clipping</h3>
<p dir="auto">Gradient clipping requires awareness that the gradients resulting from <code class="notranslate">scaler.scale(loss).backward()</code> are scaled. One simple way to account for the scale factor is by clipping to <code class="notranslate">max_norm*scaler.get_scale()</code> instead of max_norm:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = AmpScaler()
...
for input, target in data:
    optimizer.zero_grad()
    with autocast():
        output = model(input)
        loss = loss_fn(output, target)
    scaler.scale(loss).backward()

    # Gradients are scaled, so we clip to max_norm*scale
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm*scaler.get_scale())

    scaler.step(optimizer)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-v">AmpScaler</span>()
...
<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
        <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()

    <span class="pl-c"># Gradients are scaled, so we clip to max_norm*scale</span>
    <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">utils</span>.<span class="pl-en">clip_grad_norm_</span>(<span class="pl-s1">model</span>.<span class="pl-en">parameters</span>(), <span class="pl-s1">max_norm</span><span class="pl-c1">*</span><span class="pl-s1">scaler</span>.<span class="pl-en">get_scale</span>())

    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<p dir="auto">Here the scaled gradients are clipped. <code class="notranslate">scaler.step(optimizer)</code> is aware that gradients have not yet been unscaled, and unscales them under the hood before calling <code class="notranslate">optimizer.step()</code>.</p>
<h3 dir="auto">Gradient Clipping with Explicit Unscaling</h3>
<p dir="auto">The specific case of clipping scaled gradients isn’t so hard (all you have to do is clip to <code class="notranslate">max_norm*scaler.get_scale()</code>). However, in general, between the backward pass and the optimizer step you may wish to manipulate gradients in some way that’s not so easy to translate to scaled gradients. In such cases, you can unscale and step separately. Here’s how that looks, using gradient clipping as an example once more:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = AmpScaler()
...
for input, target in data:
    optimizer.zero_grad()
    with autocast():
        output = model(input)
        loss = loss_fn(output, target)
    scaler.scale(loss).backward()

    scaler.unscale(optimizer)
    # Since the optimizer's owned gradients are unscaled, we can clip to max_norm directly:
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)

    scaler.step(optimizer)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-v">AmpScaler</span>()
...
<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
        <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()

    <span class="pl-s1">scaler</span>.<span class="pl-en">unscale</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-c"># Since the optimizer's owned gradients are unscaled, we can clip to max_norm directly:</span>
    <span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-s1">utils</span>.<span class="pl-en">clip_grad_norm_</span>(<span class="pl-s1">model</span>.<span class="pl-en">parameters</span>(), <span class="pl-s1">max_norm</span>)

    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<h3 dir="auto">Gradient Penalty</h3>
<p dir="auto">(based on <strong>Higher order gradients</strong> from the <a href="https://github.com/pytorch/pytorch/releases?after=v0.3.0">release notes</a>)</p>
<p dir="auto">Gradient penalty also requires awareness that the gradients are scaled in certain places. Additionally, gradient penalty demonstrates:</p>
<ul dir="auto">
<li><code class="notranslate">torch.autograd.grad</code></li>
<li>Double-backward</li>
<li>Some clarification of what counts as a "forward pass" for the purpose of using <code class="notranslate">with autocast</code> (in other words, when exactly it's appropriate to use <code class="notranslate">with autocast</code>).</li>
</ul>
<p dir="auto">The following shows an implementation of gradient penalty under the proposed API.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = AmpScaler()
...
for input, target in data:
    optimizer.zero_grad()
    with autocast():
        output = model(input)
        loss = loss_fn(output, target)

    # We should scale outputs for the out-of-place backward pass
    grad_params = torch.autograd.grad(scaler.scale(loss), model.parameters(), create_graph=True)

    # In general, the penalty term may depend nonlinearly on the out-of-place gradients, so to be safe,
    # manually unscale them before computing the penalty.  This unscale should be autograd-exposed.
    grad_params = [p*(1./scaler.get_scale()) for p in grad_params]

    # Compute the penalty term and add it to the loss.
    # The penalty term computation is effectively another snippet of forward pass, so it makes
    # sense to enable autocasting for this section as well:
    with autocast():
        grad_norm = 0
        for grad in grad_params:
            grad_norm += grad.pow(2).sum()
        grad_norm = grad_norm.sqrt()
        loss = loss + grad_norm

    # The usual scaling for backward will now accumulate leaf gradients that are appropriately scaled.
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-v">AmpScaler</span>()
...
<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
        <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)

    <span class="pl-c"># We should scale outputs for the out-of-place backward pass</span>
    <span class="pl-s1">grad_params</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">autograd</span>.<span class="pl-en">grad</span>(<span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>), <span class="pl-s1">model</span>.<span class="pl-en">parameters</span>(), <span class="pl-s1">create_graph</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)

    <span class="pl-c"># In general, the penalty term may depend nonlinearly on the out-of-place gradients, so to be safe,</span>
    <span class="pl-c"># manually unscale them before computing the penalty.  This unscale should be autograd-exposed.</span>
    <span class="pl-s1">grad_params</span> <span class="pl-c1">=</span> [<span class="pl-s1">p</span><span class="pl-c1">*</span>(<span class="pl-c1">1.</span><span class="pl-c1">/</span><span class="pl-s1">scaler</span>.<span class="pl-en">get_scale</span>()) <span class="pl-k">for</span> <span class="pl-s1">p</span> <span class="pl-c1">in</span> <span class="pl-s1">grad_params</span>]

    <span class="pl-c"># Compute the penalty term and add it to the loss.</span>
    <span class="pl-c"># The penalty term computation is effectively another snippet of forward pass, so it makes</span>
    <span class="pl-c"># sense to enable autocasting for this section as well:</span>
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
        <span class="pl-s1">grad_norm</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>
        <span class="pl-k">for</span> <span class="pl-s1">grad</span> <span class="pl-c1">in</span> <span class="pl-s1">grad_params</span>:
            <span class="pl-s1">grad_norm</span> <span class="pl-c1">+=</span> <span class="pl-s1">grad</span>.<span class="pl-en">pow</span>(<span class="pl-c1">2</span>).<span class="pl-en">sum</span>()
        <span class="pl-s1">grad_norm</span> <span class="pl-c1">=</span> <span class="pl-s1">grad_norm</span>.<span class="pl-en">sqrt</span>()
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-s1">loss</span> <span class="pl-c1">+</span> <span class="pl-s1">grad_norm</span>

    <span class="pl-c"># The usual scaling for backward will now accumulate leaf gradients that are appropriately scaled.</span>
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()
    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<p dir="auto">Gradient penalty is a tricky case to think about but writing the code is simple once the right pattern is established.  Compared to the <a href="https://github.com/pytorch/pytorch/releases?after=v0.3.0">example in the release notes</a>, the only extra line for gradient penalty computation is the unscaling <code class="notranslate">grad_params = [p*(1./scaler.get_scale()) for p in grad_params]</code>.  I think this can be considered a documentation problem, and addressed by providing clear examples.</p>
<h3 dir="auto">Multiple Models/Optimizers/Losses</h3>
<p dir="auto">Networks must use the same <code class="notranslate">AmpScaler</code> instance (and therefore the same scale) to create gradients for all backward passes in a given iteration, otherwise we open the door to nasty corner cases.  For example, if two different losses, with different gradient scales, accumulate into the same parameters' .grads, the accumulation math breaks.  If two different losses, with different gradient scales, accumulate into different parameters owned by the same optimizer, then when you invoke <code class="notranslate">scaler.unscale(optimizer)</code>, there's no single correct value that can be used to unscale all the gradients owned by that optimizer, and handling multiple scale factors for different parameters within the same optimizer would get ugly fast.  Requiring that networks use the same <code class="notranslate">AmpScaler</code> instance for all backward passes avoids all such control flow difficulties, while still achieving what loss scaling is meant to achieve.</p>
<p dir="auto"><code class="notranslate">scaler.update()</code> must be called only at the end of the iteration, after <code class="notranslate">scaler.step(optimizer)</code> has been called for all optimizers used this iteration.  This requirement allows <code class="notranslate">update</code> to account for infs/nans found among any of the optimizers' gradients.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = torch.cuda.amp.AmpScaler()
...
for input, target in data:
    optimizer0.zero_grad()
    optimizer1.zero_grad()
    with autocast():
        output0 = model0(input)
        output1 = model1(input)
        loss0 = loss_fn(2 * output0 + 3 * output1, target)
        loss1 = loss_fn(3 * output0 - 5 * output1, target)

    scaler.scale(loss0).backward(retain_graph=True)
    scaler.scale(loss1).backward()

    # Users can choose which optimizers receive explicit unscaling
    scaler.unscale(optimizer0)

    scaler.step(optimizer0)
    scaler.step(optimizer1)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">cuda</span>.<span class="pl-s1">amp</span>.<span class="pl-v">AmpScaler</span>()
...
<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-s1">optimizer0</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-s1">optimizer1</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
        <span class="pl-s1">output0</span> <span class="pl-c1">=</span> <span class="pl-en">model0</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">output1</span> <span class="pl-c1">=</span> <span class="pl-en">model1</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">loss0</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-c1">2</span> <span class="pl-c1">*</span> <span class="pl-s1">output0</span> <span class="pl-c1">+</span> <span class="pl-c1">3</span> <span class="pl-c1">*</span> <span class="pl-s1">output1</span>, <span class="pl-s1">target</span>)
        <span class="pl-s1">loss1</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-c1">3</span> <span class="pl-c1">*</span> <span class="pl-s1">output0</span> <span class="pl-c1">-</span> <span class="pl-c1">5</span> <span class="pl-c1">*</span> <span class="pl-s1">output1</span>, <span class="pl-s1">target</span>)

    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss0</span>).<span class="pl-en">backward</span>(<span class="pl-s1">retain_graph</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss1</span>).<span class="pl-en">backward</span>()

    <span class="pl-c"># Users can choose which optimizers receive explicit unscaling</span>
    <span class="pl-s1">scaler</span>.<span class="pl-en">unscale</span>(<span class="pl-s1">optimizer0</span>)

    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer0</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer1</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<p dir="auto">I had to write Apex's Amp to handle arbitrary combinations of multiple models/optimizers/losses.  I'm painfully aware of the complicated combinations of models/optimizers/losses people want to implement.  In my opinion, the proposed interface permits a great deal of flexibility in network design.</p>
<h3 dir="auto">Gradient accumulation</h3>
<p dir="auto">Gradient accumulation across iterations (between steps) is a common use case.  The proposed API accommodates gradient accumulation without trouble:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = AmpScaler()
...
for i, (input, target) in enumerate(data):
    with autocast():
        output = model(input)
        loss = loss_fn(output, target)
        loss = loss/iters_to_accumulate
    scaler.scale(loss).backward()
    if (i + 1) % iters_to_accumulate == 0:
        scaler.step(optimizer)
        scaler.update()
        optimizer.zero_grad()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-v">AmpScaler</span>()
...
<span class="pl-k">for</span> <span class="pl-s1">i</span>, (<span class="pl-s1">input</span>, <span class="pl-s1">target</span>) <span class="pl-c1">in</span> <span class="pl-en">enumerate</span>(<span class="pl-s1">data</span>):
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
        <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-s1">loss</span><span class="pl-c1">/</span><span class="pl-s1">iters_to_accumulate</span>
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()
    <span class="pl-k">if</span> (<span class="pl-s1">i</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>) <span class="pl-c1">%</span> <span class="pl-s1">iters_to_accumulate</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span>:
        <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
        <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()
        <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()</pre></div>
<h3 dir="auto">Switching automatic mixed precision on and off</h3>
<p dir="auto">If users want to run with or without autocasting+gradient scaling, they shouldn't have to litter their code with if statements.  The API should allow one code path that accommodates easily switching autocasting+gradient scaling on and off.</p>
<p dir="auto">The autocasting context manager and <code class="notranslate">AmpScaler</code> constructor provide such convenience by accepting an <code class="notranslate">enabled=False</code> argument.</p>
<p dir="auto">In the following example, autocasting and gradient scaling can be switched on and off by flipping <code class="notranslate">args.use_mixed_precision</code> with no additional control flow required.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = AmpScaler(enabled=args.use_mixed_precision)
...
for input, target in data:
    optimizer.zero_grad()
    with autocast(enabled=args.use_mixed_precision):
        output = model(input)
        loss = loss_fn(output, target)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-v">AmpScaler</span>(<span class="pl-s1">enabled</span><span class="pl-c1">=</span><span class="pl-s1">args</span>.<span class="pl-s1">use_mixed_precision</span>)
...
<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()
    <span class="pl-k">with</span> <span class="pl-en">autocast</span>(<span class="pl-s1">enabled</span><span class="pl-c1">=</span><span class="pl-s1">args</span>.<span class="pl-s1">use_mixed_precision</span>):
        <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()
    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<h3 dir="auto">Batch replay</h3>
<p dir="auto">Sometimes every iteration/batch is valuable enough that users don't want to skip any.  Instead, it's preferable to replay the batch with a reduced loss scale until gradients do not contain infs/nans.  Batch replay control flow is not provided by the API alone, but with the proposed gradient scaling PR, it would be easy to rig:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="scaler = AmpScaler()
...
for input, target in data:
    # Replay the batch, updating the scale if necessary, until we receive gradients that aren't inf/nan.
    while True:
        optimizer.zero_grad()
        with autocast():
            output = model(input)
            loss = loss_fn(output, target)
        scaler.scale(loss).backward()
        scaler.unscale(optimizer)
        if scaler._found_inf(optimizer).item():
            scaler.update()
        else:
            break
    scaler.step(optimizer)
    scaler.update()"><pre class="notranslate"><span class="pl-s1">scaler</span> <span class="pl-c1">=</span> <span class="pl-v">AmpScaler</span>()
...
<span class="pl-k">for</span> <span class="pl-s1">input</span>, <span class="pl-s1">target</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-c"># Replay the batch, updating the scale if necessary, until we receive gradients that aren't inf/nan.</span>
    <span class="pl-k">while</span> <span class="pl-c1">True</span>:
        <span class="pl-s1">optimizer</span>.<span class="pl-en">zero_grad</span>()
        <span class="pl-k">with</span> <span class="pl-en">autocast</span>():
            <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input</span>)
            <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fn</span>(<span class="pl-s1">output</span>, <span class="pl-s1">target</span>)
        <span class="pl-s1">scaler</span>.<span class="pl-en">scale</span>(<span class="pl-s1">loss</span>).<span class="pl-en">backward</span>()
        <span class="pl-s1">scaler</span>.<span class="pl-en">unscale</span>(<span class="pl-s1">optimizer</span>)
        <span class="pl-k">if</span> <span class="pl-s1">scaler</span>.<span class="pl-en">_found_inf</span>(<span class="pl-s1">optimizer</span>).<span class="pl-en">item</span>():
            <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()
        <span class="pl-k">else</span>:
            <span class="pl-k">break</span>
    <span class="pl-s1">scaler</span>.<span class="pl-en">step</span>(<span class="pl-s1">optimizer</span>)
    <span class="pl-s1">scaler</span>.<span class="pl-en">update</span>()</pre></div>
<h1 dir="auto">Alternatives</h1>
<h3 dir="auto">Python-side alternatives for gradient scaling and unscaling</h3>
<p dir="auto">The supplementary information contains an in-depth discussion of some alternatives I considered for the <a href="https://gist.github.com/mcarilli/43445260404c8d7cd79d84439808250e#file-scaling_apis_considered-md">gradient scaling</a> and <a href="https://gist.github.com/mcarilli/43445260404c8d7cd79d84439808250e#file-unscaling_apis_considered-md">gradient unscaling</a> API.</p>
<h3 dir="auto">Gradient scaling in the autograd backend</h3>
<p dir="auto">I recently submitted <a href="https://github.com/pytorch/pytorch/pull/24893" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/24893/hovercard">a PR</a> that implemented gradient scaling directly in the autograd engine (Engine::execute).</p>
<p dir="auto">Benefits:</p>
<ul dir="auto">
<li>It automatically enabled gradient scaling for all typical single- and double-backward use cases (including complex cases like gradient penalties) without requiring any change to user scripts.</li>
<li>Gradients returned visibly to the user, either in-place via <code class="notranslate">.grad</code> attributes or out-of-place via a call to <code class="notranslate">torch.autograd.grad</code>, were only ever unscaled, eliminating the need to change gradient clipping or manually unscale before computing gradient penalties.</li>
</ul>
<p dir="auto">Drawbacks:</p>
<ul dir="auto">
<li>Modifying engine.cpp, especially for a GPU-specific purpose, is not to be done lightly (ie, not until alternatives are exhausted).</li>
<li>It's unclear what the user-facing API would look like.  I figured the implementation was general enough to permit many options, and the exact API could be nailed down later.  It certainly requires maintaining a global "amp state."</li>
<li>Altering the backend to do black-box gradient scaling and require no change to user scripts is a double-edged sword.  Explicit Python-side control and visibility of gradient scaling, as we propose above, is not a bad thing.  <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cbcase/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cbcase">@cbcase</a>, this seems like an instance of the <a href="https://www.jwz.org/doc/worse-is-better.html" rel="nofollow">https://www.jwz.org/doc/worse-is-better.html</a> thing you told me about...</li>
</ul>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vincentqb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vincentqb">@vincentqb</a></p>
<p dir="auto">Hello I have an error with the command<br>
MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install<br>
I'm trying to install Pytorch for Mac I’m following this tutorial :<br>
<a href="https://www.cs.rochester.edu/u/kautz/Installing-Pytorch-Cuda-on-Macbook.html" rel="nofollow">https://www.cs.rochester.edu/u/kautz/Installing-Pytorch-Cuda-on-Macbook.html</a></p>
<p dir="auto">I’m using macOS 10.13.4<br>
I have installed:</p>
<ul dir="auto">
<li>xcode 8.3.3</li>
<li>Python 3.6.7 with anaconda</li>
<li>Cuda 9.0</li>
<li>Install Cuda driver 396<br>
I was supposed to install cuDNN v7.1.4 for cuda 9.0 but this is no longer available for Mac so I took  cuDNN v7.0.4 Library for OSX</li>
</ul>
<p dir="auto">I installed pytorch with the following command:<br>
conda install pytorch=0.4.1 -c pytorch<br>
I did that because conda install pytorch=0.4.1 cuda90 -c pytorch did not work. I think the whl file is not longer available on <a href="https://pytorch.org/get-started/previous-versions/" rel="nofollow">https://pytorch.org/get-started/previous-versions/</a><br>
So after I did that, I ran the command line MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install<br>
Now I get this :</p>
<p dir="auto">running install<br>
running build_deps</p>
<ul dir="auto">
<li>USE_CUDA=0</li>
<li>USE_ROCM=0</li>
<li>USE_NNPACK=0</li>
<li>USE_MKLDNN=0</li>
<li>USE_GLOO_IBVERBS=0</li>
<li>USE_DISTRIBUTED_MW=0</li>
<li>FULL_CAFFE2=0</li>
<li>[[ 6 -gt 0 ]]</li>
<li>case "$1" in</li>
<li>USE_CUDA=1</li>
<li>shift</li>
<li>[[ 5 -gt 0 ]]</li>
<li>case "$1" in</li>
<li>USE_NNPACK=1</li>
<li>shift</li>
<li>[[ 4 -gt 0 ]]</li>
<li>case "$1" in</li>
<li>break</li>
<li>CMAKE_INSTALL='make install'</li>
<li>USER_CFLAGS=</li>
<li>USER_LDFLAGS=</li>
<li>[[ -n '' ]]</li>
<li>[[ -n '' ]]</li>
<li>[[ -n '' ]]<br>
++ uname</li>
<li>'[' Darwin == Darwin ']'</li>
<li>'[' -d /usr/local/opt/ccache/libexec ']'<br>
++ dirname tools/build_pytorch_libs.sh</li>
<li>cd tools/..<br>
+++ pwd<br>
++ printf '%q\n' /Users/demanouwamo/Downloads/pytorch</li>
<li>PWD=/Users/demanouwamo/Downloads/pytorch</li>
<li>BASE_DIR=/Users/demanouwamo/Downloads/pytorch</li>
<li>TORCH_LIB_DIR=/Users/demanouwamo/Downloads/pytorch/torch/lib</li>
<li>INSTALL_DIR=/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install</li>
<li>THIRD_PARTY_DIR=/Users/demanouwamo/Downloads/pytorch/third_party</li>
<li>CMAKE_VERSION=cmake</li>
<li>C_FLAGS=' -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/TH" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THC"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THS" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THCS"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THNN" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THCUNN"'</li>
<li>C_FLAGS=' -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/TH" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THC"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THS" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THCS"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THNN" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THCUNN" -DOMPI_SKIP_MPICXX=1'</li>
<li>LDFLAGS='-L"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/lib" '</li>
<li>LD_POSTFIX=.so<br>
++ uname</li>
<li>[[ Darwin == \D\a\r\w\i\n ]]</li>
<li>LDFLAGS='-L"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/lib"  -Wl,-rpath,@loader_path'</li>
<li>LD_POSTFIX=.dylib</li>
<li>CPP_FLAGS=' -std=c++11 '</li>
<li>GLOO_FLAGS=</li>
<li>THD_FLAGS=</li>
<li>NCCL_ROOT_DIR=/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install</li>
<li>[[ 1 -eq 1 ]]</li>
<li>GLOO_FLAGS='-DUSE_CUDA=1 -DNCCL_ROOT_DIR=/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install'</li>
<li>[[ 0 -eq 1 ]]</li>
<li>[[ 0 -eq 1 ]]</li>
<li>CWRAP_FILES='/Users/demanouwamo/Downloads/pytorch/torch/lib/ATen/Declarations.cwrap;/Users/demanouwamo/Downloads/pytorch/torch/lib/THNN/generic/THNN.h;/Users/demanouwamo/Downloads/pytorch/torch/lib/THCUNN/generic/THCUNN.h;/Users/demanouwamo/Downloads/pytorch/torch/lib/ATen/nn.yaml'</li>
<li>CUDA_NVCC_FLAGS=' -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/TH" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THC"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THS" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THCS"   -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THNN" -I"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/include/THCUNN" -DOMPI_SKIP_MPICXX=1'</li>
<li>[[ -z '' ]]</li>
<li>CUDA_DEVICE_DEBUG=0</li>
<li>'[' -z 4 ']'</li>
<li>BUILD_TYPE=Release</li>
<li>[[ -n '' ]]</li>
<li>[[ -n '' ]]</li>
<li>echo 'Building in Release mode'<br>
Building in Release mode</li>
<li>mkdir -p torch/lib/tmp_install</li>
<li>for arg in '"$@"'</li>
<li>[[ caffe2 == \n\c\c\l ]]</li>
<li>[[ caffe2 == \g\l\o\o ]]</li>
<li>[[ caffe2 == \c\a\f\f\e\2 ]]</li>
<li>pushd /Users/demanouwamo/Downloads/pytorch<br>
~/Downloads/pytorch ~/Downloads/pytorch</li>
<li>build_caffe2</li>
<li>[[ -z '' ]]</li>
<li>EXTRA_CAFFE2_CMAKE_FLAGS=()</li>
<li>[[ -n '' ]]</li>
<li>[[ -n /anaconda3/envs/ptc/lib/python3.6/site-packages ]]</li>
<li>EXTRA_CAFFE2_CMAKE_FLAGS+=("-DCMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH")</li>
<li>mkdir -p build</li>
<li>pushd build<br>
~/Downloads/pytorch/build ~/Downloads/pytorch ~/Downloads/pytorch</li>
<li>cmake .. -DBUILDING_WITH_TORCH_LIBS=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_CAFFE2=0 -DBUILD_ATEN=ON -DBUILD_PYTHON=0 -DBUILD_BINARY=OFF -DBUILD_SHARED_LIBS=ON -DONNX_NAMESPACE=onnx_torch -DUSE_CUDA=1 -DCAFFE2_STATIC_LINK_CUDA= -DUSE_ROCM=0 -DUSE_NNPACK=1 -DCUDNN_INCLUDE_DIR=/usr/local/cuda/include -DCUDNN_LIB_DIR=/usr/local/cuda/lib -DCUDNN_LIBRARY=/usr/local/cuda/lib/libcudnn.7.dylib -DUSE_MKLDNN=0 -DMKLDNN_INCLUDE_DIR= -DMKLDNN_LIB_DIR= -DMKLDNN_LIBRARY= -DCMAKE_INSTALL_PREFIX=/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCMAKE_C_FLAGS= -DCMAKE_CXX_FLAGS= '-DCMAKE_EXE_LINKER_FLAGS=-L"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/lib"  -Wl,-rpath,@loader_path ' '-DCMAKE_SHARED_LINKER_FLAGS=-L"/Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install/lib"  -Wl,-rpath,@loader_path ' -DCMAKE_PREFIX_PATH=/anaconda3/envs/ptc/lib/python3.6/site-packages<br>
-- CLANG_VERSION_STRING:         8.1<br>
-- Need to define long as a separate typeid.<br>
-- std::exception_ptr is supported.<br>
-- NUMA is not available<br>
-- Turning off deprecation warning due to glog.<br>
-- Current compiler supports avx2 extention. Will build perfkernels.<br>
-- Building using own protobuf under third_party per request.<br>
-- Use custom protobuf build.<br>
-- Caffe2 protobuf include directory: <math-renderer class="js-inline-math" style="display: inline" data-static-url="https://github.githubassets.com/static" data-run-id="c5471ae686d1da39757efcc18e350543">$&amp;lt;BUILD_INTERFACE:/Users/demanouwamo/Downloads/pytorch/third_party/protobuf/src&amp;gt;$</math-renderer>&amp;lt;INSTALL_INTERFACE:include&gt;<br>
-- The BLAS backend of choice:MKL<br>
-- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m]<br>
--   Library mkl_intel_lp64: /anaconda3/envs/ptc/lib/libmkl_intel_lp64.dylib<br>
--   Library mkl_intel_thread: /anaconda3/envs/ptc/lib/libmkl_intel_thread.dylib<br>
--   Library mkl_core: /anaconda3/envs/ptc/lib/libmkl_core.dylib<br>
--   Library iomp5: /anaconda3/envs/ptc/lib/libiomp5.dylib<br>
--   Library pthread: /usr/lib/libpthread.dylib<br>
--   Library m: /usr/lib/libm.dylib<br>
-- Brace yourself, we are building NNPACK<br>
-- Found system Eigen at /usr/local/include/eigen3<br>
-- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR)<br>
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "7.0")<br>
-- Caffe2: CUDA detected: 9.0<br>
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc<br>
-- Caffe2: CUDA toolkit directory: /usr/local/cuda<br>
-- Caffe2: Header version is: 9.0<br>
-- Found cuDNN: v7.0.4  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib/libcudnn.7.dylib)<br>
-- Automatic GPU detection failed. Building for common architectures.<br>
-- Autodetected CUDA architecture(s): 3.0;3.5;5.0;5.2;6.0;6.1;7.0;7.0+PTX<br>
-- Added CUDA NVCC flags for: -gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_70,code=compute_70<br>
-- Could NOT find CUB (missing: CUB_INCLUDE_DIR)<br>
-- Found CUDA with FP16 support, compiling with torch.cuda.HalfTensor<br>
-- Removing -DNDEBUG from compile flags<br>
-- MAGMA not found. Compiling without MAGMA support<br>
-- Could not find hardware support for NEON on this machine.<br>
-- No OMAP3 processor on this machine.<br>
-- No OMAP4 processor on this machine.<br>
-- SSE2 Found<br>
-- SSE3 Found<br>
-- AVX Found<br>
-- AVX2 Found<br>
-- Atomics: using GCC intrinsics<br>
-- Found a library with BLAS API (mkl).<br>
-- Found a library with LAPACK API. (mkl)<br>
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "5.5")<br>
-- Could NOT find MKLDNN (missing: MKLDNN_INCLUDE_DIR MKLDNN_LIBRARY)<br>
-- MKLDNN not found. Compiling without MKLDNN support<br>
-- Using python found in /anaconda3/envs/ptc/bin/python<br>
-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)<br>
-- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)<br>
-- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)<br>
-- Configuring build for SLEEF-v3.2<br>
Target system: Darwin-17.6.0<br>
Target processor: x86_64<br>
Host system: Darwin-17.6.0<br>
Host processor: x86_64<br>
Detected C compiler: AppleClang @ /Users/demanouwamo/Downloads/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang<br>
-- Using option <code class="notranslate">-Wall -Wno-unused -Wno-attributes -Wno-unused-result -ffp-contract=off -fno-math-errno -fno-trapping-math</code> to compile libsleef<br>
-- Building shared libs : OFF<br>
-- MPFR : /usr/local/lib/libmpfr.dylib<br>
-- MPFR header file in /usr/local/include<br>
-- GMP : /usr/local/lib/libgmp.dylib<br>
-- RUNNING_ON_TRAVIS : 0<br>
-- COMPILER_SUPPORTS_OPENMP :<br>
CMake Warning at CMakeLists.txt:346 (message):<br>
Generated cmake files are only fully tested if one builds with system glog,<br>
gflags, and protobuf.  Other settings may generate files that are not well<br>
tested.</li>
</ul>
<h2 dir="auto">--<br>
-- ******** Summary ********<br>
-- General:<br>
--   CMake version         : 3.12.2<br>
--   CMake command         : /anaconda3/envs/ptc/bin/cmake<br>
--   Git version           :<br>
--   System                : Darwin<br>
--   C++ compiler          : /Users/demanouwamo/Downloads/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++<br>
--   C++ compiler version  : 8.1.0.8020042<br>
--   BLAS                  : MKL<br>
--   CXX flags             :   -msse4.1 -msse4.2   --std=c++11  -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DONNX_NAMESPACE=onnx_torch -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-aliasing -Wno-error=deprecated-declarations<br>
--   Build type            : Release<br>
--   Compile definitions   : USE_GCC_ATOMICS=1;TH_BLAS_MKL;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1<br>
--   CMAKE_PREFIX_PATH     : /anaconda3/envs/ptc/lib/python3.6/site-packages<br>
--   CMAKE_INSTALL_PREFIX  : /Users/demanouwamo/Downloads/pytorch/torch/lib/tmp_install</h2>
<p dir="auto">--   BUILD_CAFFE2          : 0<br>
--   BUILD_ATEN            : ON<br>
--   BUILD_BINARY          : OFF<br>
--   BUILD_CUSTOM_PROTOBUF : ON<br>
--     Link local protobuf : ON<br>
--   BUILD_PYTHON          : 0<br>
--   BUILD_SHARED_LIBS     : ON<br>
--   BUILD_TEST            : OFF<br>
--   USE_ASAN              : OFF<br>
--   USE_ATEN              : OFF<br>
--   USE_CUDA              : 1<br>
--     CUDA static link    :<br>
--     USE_CUDNN           : ON<br>
--     CUDA version        : 9.0<br>
--     cuDNN version       : 7.0.4<br>
--     CUDA root directory : /usr/local/cuda<br>
--     CUDA library        : /usr/local/cuda/lib/libcuda.dylib<br>
--     cudart library      : /usr/local/cuda/lib/libcudart_static.a;-Wl,-rpath,/usr/local/cuda/lib<br>
--     cublas library      : /usr/local/cuda/lib/libcublas.dylib;/usr/local/cuda/lib/libcublas_device.a<br>
--     cufft library       : /usr/local/cuda/lib/libcufft.dylib<br>
--     curand library      : /usr/local/cuda/lib/libcurand.dylib<br>
--     cuDNN library       : /usr/local/cuda/lib/libcudnn.7.dylib<br>
--     nvrtc               : /usr/local/cuda/lib/libnvrtc.dylib<br>
--     CUDA include path   : /usr/local/cuda/include<br>
--     NVCC executable     : /usr/local/cuda/bin/nvcc<br>
--     CUDA host compiler  : /Users/demanouwamo/Downloads/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang<br>
--     USE_TENSORRT        : OFF<br>
--   USE_ROCM              : OFF<br>
--   USE_EIGEN_FOR_BLAS    :<br>
--   USE_FFMPEG            : OFF<br>
--   USE_GFLAGS            : OFF<br>
--   USE_GLOG              : OFF<br>
--   USE_GLOO              : OFF<br>
--   USE_LEVELDB           : OFF<br>
--   USE_LITE_PROTO        : OFF<br>
--   USE_LMDB              : OFF<br>
--   USE_METAL             : OFF<br>
--   USE_MKL               :<br>
--   USE_MOBILE_OPENGL     : OFF<br>
--   USE_MPI               : OFF<br>
--   USE_NCCL              : OFF<br>
--   USE_NERVANA_GPU       : OFF<br>
--   USE_NNPACK            : 1<br>
--   USE_OBSERVERS         : OFF<br>
--   USE_OPENCL            : OFF<br>
--   USE_OPENCV            : OFF<br>
--   USE_OPENMP            : OFF<br>
--   USE_PROF              : OFF<br>
--   USE_REDIS             : OFF<br>
--   USE_ROCKSDB           : OFF<br>
--   USE_ZMQ               : OFF<br>
--   USE_DISTRIBUTED       : OFF<br>
--   Public Dependencies  : /anaconda3/envs/ptc/lib/libmkl_intel_lp64.dylib;/anaconda3/envs/ptc/lib/libmkl_intel_thread.dylib;/anaconda3/envs/ptc/lib/libmkl_core.dylib;/anaconda3/envs/ptc/lib/libiomp5.dylib;/usr/lib/libpthread.dylib;/usr/lib/libm.dylib<br>
--   Private Dependencies : nnpack;cpuinfo;onnxifi_loader<br>
-- Configuring done<br>
-- Generating done<br>
-- Build files have been written to: /Users/demanouwamo/Downloads/pytorch/build</p>
<ul dir="auto">
<li>make install -j4<br>
[  1%] Built target js_embed<br>
[  2%] Built target pthreadpool<br>
[  6%] Built target libprotobuf-lite<br>
[  8%] Built target cpuinfo<br>
[  8%] Built target onnxifi_loader<br>
[  9%] Built target common<br>
[  9%] Built target mkrename<br>
[ 20%] Built target libprotobuf<br>
[ 21%] Built target mkdisp<br>
[ 21%] Built target mkalias<br>
[ 21%] Built target ATEN_CPU_FILES_GEN_TARGET<br>
[ 21%] Built target ATEN_CUDA_FILES_GEN_TARGET<br>
[ 21%] Built target mkmasked_gnuabi<br>
[ 21%] Built target mkrename_gnuabi<br>
[ 21%] Built target arraymap<br>
[ 22%] Built target nnpack_reference_layers<br>
[ 26%] Built target nnpack<br>
[ 27%] Built target headers<br>
[ 27%] Built target renamedsp256.h_generated<br>
[ 27%] Built target dispavx.c_generated<br>
[ 28%] Built target renameSSE2.h_generated<br>
[ 28%] Built target renameAVX.h_generated<br>
[ 28%] Built target renameFMA4.h_generated<br>
[ 28%] Built target renameSSE4.h_generated<br>
[ 28%] Built target renameAVX2128.h_generated<br>
[ 41%] Built target libprotoc<br>
[ 41%] Built target renameAVX2.h_generated<br>
[ 41%] Built target dispsse.c_generated<br>
[ 41%] Built target renamedsp128.h_generated<br>
[ 41%] Built target renameAVX512F.h_generated<br>
[ 41%] Generating alias_avx512f.h<br>
[ 41%] Built target caffe2_protos<br>
[ 41%] Built target protoc<br>
[ 41%] Built target dispavx_obj<br>
[ 41%] Built target alias_avx512f.h_generated<br>
[ 41%] Built target sleefsse2<br>
[ 41%] Built target sleefavx<br>
[ 42%] Built target sleeffma4<br>
[ 43%] Built target sleefsse4<br>
[ 44%] Built target sleefavx2128<br>
[ 44%] Built target sleefavx2<br>
[ 45%] Built target dispsse_obj<br>
[ 45%] Built target sleefavx512f<br>
[ 45%] Built target gen_onnx_proto<br>
[ 46%] Built target sleef<br>
[ 47%] Built target onnx_proto<br>
[ 51%] Built target onnx<br>
[ 69%] Built target caffe2<br>
[ 69%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/<strong>/aten/src/THC/caffe2_gpu_generated_THCSleep.cu.o<br>
[ 69%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/</strong>/aten/src/THC/caffe2_gpu_generated_THCReduceApplyUtils.cu.o<br>
[ 69%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/<strong>/aten/src/THC/caffe2_gpu_generated_THCBlas.cu.o<br>
[ 69%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/</strong>/aten/src/THC/caffe2_gpu_generated_THCStorage.cu.o<br>
nvcc fatal   : Unsupported gpu architecture 'compute_70'<br>
nvcc fatal   : Unsupported gpu architecture 'compute_70'<br>
CMake Error at caffe2_gpu_generated_THCSleep.cu.o.Release.cmake:219 (message):<br>
Error generating<br>
/Users/demanouwamo/Downloads/pytorch/build/caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/./caffe2_gpu_generated_THCSleep.cu.o</li>
</ul>
<p dir="auto">nvcc fatal   : Unsupported gpu architecture 'compute_70'<br>
CMake Error at caffe2_gpu_generated_THCBlas.cu.o.Release.cmake:219 (message):<br>
Error generating<br>
/Users/demanouwamo/Downloads/pytorch/build/caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/./caffe2_gpu_generated_THCBlas.cu.o</p>
<p dir="auto">CMake Error at caffe2_gpu_generated_THCReduceApplyUtils.cu.o.Release.cmake:219 (message):<br>
Error generating<br>
/Users/demanouwamo/Downloads/pytorch/build/caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/./caffe2_gpu_generated_THCReduceApplyUtils.cu.o</p>
<p dir="auto">make[2]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/<strong>/aten/src/THC/caffe2_gpu_generated_THCSleep.cu.o] Error 1<br>
make[2]: *** Waiting for unfinished jobs....<br>
make[2]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/</strong>/aten/src/THC/caffe2_gpu_generated_THCBlas.cu.o] Error 1<br>
make[2]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/<strong>/aten/src/THC/caffe2_gpu_generated_THCReduceApplyUtils.cu.o] Error 1<br>
nvcc fatal   : Unsupported gpu architecture 'compute_70'<br>
CMake Error at caffe2_gpu_generated_THCStorage.cu.o.Release.cmake:219 (message):<br>
Error generating<br>
/Users/demanouwamo/Downloads/pytorch/build/caffe2/CMakeFiles/caffe2_gpu.dir/</strong>/aten/src/THC/./caffe2_gpu_generated_THCStorage.cu.o</p>
<p dir="auto">make[2]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/caffe2_gpu_generated_THCStorage.cu.o] Error 1<br>
make[1]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/all] Error 2<br>
make: *** [all] Error 2<br>
Failed to run 'bash tools/build_pytorch_libs.sh --use-cuda --use-nnpack caffe2 nanopb libshm THD</p>
<p dir="auto">Could you help me please.</p>