<p dir="auto">We define 4D tensor as stored in channels last memory format, when dimensions order is NCHW and <code class="notranslate">C-strides &lt; W-strides &lt; H-strides &lt; N-strides</code> <em>(If size of any dimension is equal to 1, this dimension strides value is not taken into account)</em>.</p>
<p dir="auto">Channels last contiguous tensor is channel last tensor which occupies contiguous memory block. So <code class="notranslate">x.is_contiguous(memory_format=torch.channels_last)</code> checks if tensor is channels last contiguous.</p>
<p dir="auto">The goal of the experiment is to use channels last memory format in all Resnet (<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py">https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py</a>) model's operators and to measure performance gains on the Volta devices with CudNN library available.</p>
<p dir="auto">This experiment requires:</p>
<ol dir="auto">
<li>Update operators kernels to follow the next rule: if one of the operator's inputs is channel last tensor, all outputs should also be in the channel last memory format.</li>
<li>For better performance gain, update DataLoader to output channel last tensors.</li>
</ol>
<p dir="auto">To avoid changing the model itself and more importantly, introduce this optimization to the existing saved models. We need to introduce the next changes:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">to</code> operator should preserve memory format. <code class="notranslate">copy_device_to_device</code> should be memory format aware. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="477596984" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/23899" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/23899/hovercard" href="https://github.com/pytorch/pytorch/pull/23899">#23899</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">empty_like</code> operator should preserve memory format by default. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="477596984" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/23899" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/23899/hovercard" href="https://github.com/pytorch/pytorch/pull/23899">#23899</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">resize_as_</code> operator should be memory format aware. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="477596984" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/23899" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/23899/hovercard" href="https://github.com/pytorch/pytorch/pull/23899">#23899</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> <code class="notranslate">clone</code> operator should preserve memory format. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="477596984" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/23899" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/23899/hovercard" href="https://github.com/pytorch/pytorch/pull/23899">#23899</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">scatter</code> and <code class="notranslate">gather</code> functions should be memory format aware. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="479168416" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/24121" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/24121/hovercard" href="https://github.com/pytorch/pytorch/pull/24121">#24121</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">TensorIterator</code> based point-wise operators should preserve memory format (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="478700314" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/24038" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/24038/hovercard" href="https://github.com/pytorch/pytorch/pull/24038">#24038</a>).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">adaptive_avg_pool2d_cuda</code> and <code class="notranslate">adaptive_avg_pool2d_backward_cuda</code> should have channel last optimized kernels. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="481041278" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/24396" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/24396/hovercard" href="https://github.com/pytorch/pytorch/pull/24396">#24396</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> <code class="notranslate">max_pool2d_with_indices_cuda</code> and <code class="notranslate">max_pool2d_with_indices_backward_cuda</code> should have channel last optimized kernels (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="482565934" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/24872" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/24872/hovercard" href="https://github.com/pytorch/pytorch/pull/24872">#24872</a>).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">cudnn_batch_norm</code> and <code class="notranslate">cudnn_batch_norm_backward</code> should support channels last memory format. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="477240072" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/23861" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/23861/hovercard" href="https://github.com/pytorch/pytorch/pull/23861">#23861</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> <code class="notranslate">cudnn_convolution_forward</code> and <code class="notranslate">cudnn_convolution_backward</code> should support channels last memory format. (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="477240072" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/23861" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/23861/hovercard" href="https://github.com/pytorch/pytorch/pull/23861">#23861</a>)</li>
</ul>
<p dir="auto">Writing memory format aware operators require special functions introduced in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="473032826" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/23391" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/23391/hovercard" href="https://github.com/pytorch/pytorch/pull/23391">#23391</a></p>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="auto memory_format = input_tensor.suggest_memory_format();
auto output_tensor = at::empty(output_shape, memory_format);

switch (memory_format) {
  case MemoryFormat::ChannelsLast: {
    input_cl_contiguous = input_tensor.contiguous(
        MemoryFormat::ChannelsLast); // if kernel requires memory contiguous
                                     // tensor
    // .... kernel code
    break;
  }
  case MemoryFormat::Contiguous: {
    // .... standard kernel
    break;
  }
  default:
    TORCH_CHECK(
        false,
        &quot;Unsupported memory format. Supports only ChannelsLast, Contiguous&quot;);
}"><pre class="notranslate"><span class="pl-k">auto</span> memory_format = input_tensor.suggest_memory_format();
<span class="pl-k">auto</span> output_tensor = at::empty(output_shape, memory_format);

<span class="pl-k">switch</span> (memory_format) {
  <span class="pl-k">case</span> MemoryFormat::ChannelsLast: {
    input_cl_contiguous = input_tensor.<span class="pl-c1">contiguous</span>(
        MemoryFormat::ChannelsLast); <span class="pl-c"><span class="pl-c">//</span> if kernel requires memory contiguous</span>
                                     <span class="pl-c"><span class="pl-c">//</span> tensor</span>
    <span class="pl-c"><span class="pl-c">//</span> .... kernel code</span>
    <span class="pl-k">break</span>;
  }
  <span class="pl-k">case</span> MemoryFormat::Contiguous: {
    <span class="pl-c"><span class="pl-c">//</span> .... standard kernel</span>
    <span class="pl-k">break</span>;
  }
  <span class="pl-k">default</span>:
    <span class="pl-c1">TORCH_CHECK</span>(
        <span class="pl-c1">false</span>,
        <span class="pl-s"><span class="pl-pds">"</span>Unsupported memory format. Supports only ChannelsLast, Contiguous<span class="pl-pds">"</span></span>);
}</pre></div>
<h2 dir="auto">Notes</h2>
<ul dir="auto">
<li>Resnet calls <code class="notranslate">x = x.reshape(x.size(0), -1)</code> before linear layers, we are going to update <code class="notranslate">reshape</code> and <code class="notranslate">view</code> code and convert tensor's memory format to <code class="notranslate">torch.contiguous_format</code> at this step.</li>
<li>Making old models/libraries to work faster also requires <strong>BC sacrifices</strong>, such as<br>
<code class="notranslate">empty_like</code> ( and all _like operators ) will return channels last tensor if input is channels last, similar will apply to <code class="notranslate">to</code>, <code class="notranslate">clone</code>, <code class="notranslate">resize_as</code>. We are thinking about the ability to control suggest_memory_format behaviors by the global variable.</li>
</ul>
<p dir="auto">If you have a question or would like help and support, please ask at our<br>
<a href="https://discuss.pytorch.org/" rel="nofollow">forums</a>.</p>
<p dir="auto">If you are submitting a feature request, please preface the title with [feature request].<br>
If you are submitting a bug report, please fill in the following details.</p>
<h2 dir="auto">Issue description</h2>
<p dir="auto">Unable to do git clone 0.4.0 of pytorch<br>
Provide a short description.</p>
<h2 dir="auto">Code example</h2>
<p dir="auto">Cloning into 'pytorch-src'...<br>
remote: Counting objects: 101473, done.<br>
remote: Compressing objects: 100% (62/62), done.<br>
remote: Total 101473 (delta 24), reused 20 (delta 10), pack-reused 101401<br>
Receiving objects: 100% (101473/101473), 54.60 MiB | 49.09 MiB/s, done.<br>
Resolving deltas: 100% (75514/75514), done.<br>
~/pytorch/pytorch-src ~/pytorch<br>
Note: checking out 'v0.4.0'.</p>
<p dir="auto">You are in 'detached HEAD' state. You can look around, make experimental<br>
changes and commit them, and you can discard any commits you make in this<br>
state without impacting any branches by performing another checkout.</p>
<p dir="auto">If you want to create a new branch to retain commits you create, you may<br>
do so (now or later) by using -b with the checkout command again. Example:</p>
<p dir="auto">git checkout -b <br>
.....<br>
Cloning into '/home/ec2-user/pytorch/pytorch-src/third_party/eigen'...<br>
Username for '<a href="https://github.com">https://github.com</a>': gautamkmr<br>
Password for '<a href="https://gautamkmr@github.com">https://gautamkmr@github.com</a>':<br>
remote: Repository not found.<br>
fatal: repository '<a href="https://github.com/RLovelett/eigen.git/">https://github.com/RLovelett/eigen.git/</a>' not found<br>
fatal: clone of '<a href="https://github.com/RLovelett/eigen.git">https://github.com/RLovelett/eigen.git</a>' into submodule path '/home/ec2-user/pytorch/pytorch-src/third_party/eigen' failed<br>
Failed to clone 'third_party/eigen' a second time, aborting<br>
[ec2-user@ip-172-31-17-149 pytorch]$</p>
<h2 dir="auto">System Info</h2>
<p dir="auto">Please copy and paste the output from our<br>
<a href="https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py" rel="nofollow">environment collection script</a><br>
(or fill out the checklist below manually).</p>
<p dir="auto">You can get the script and run it with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py"><pre class="notranslate"><code class="notranslate">wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
</code></pre></div>
<ul dir="auto">
<li>PyTorch or Caffe2:</li>
<li>How you installed PyTorch (conda, pip, source):</li>
<li>Build command you used (if compiling from source):</li>
<li>OS:</li>
<li>PyTorch version:</li>
<li>Python version:</li>
<li>CUDA/cuDNN version:</li>
<li>GPU models and configuration:</li>
<li>GCC version (if compiling from source):</li>
<li>CMake version:</li>
<li>Versions of any other relevant libraries:</li>
</ul>