<h2 dir="auto">"type id is VariableTensorId" runtime error for some mobile models</h2>
<p dir="auto">guard torch::jit::load() with AutoGradMode &amp; AutoNonVariableTypeMode</p>
<p dir="auto">Summary:<br>
Since PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="488760465" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/25597" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/25597/hovercard" href="https://github.com/pytorch/pytorch/pull/25597">#25597</a> landed some models started failing to run with mobile<br>
build with the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="type id is VariableTensorId
terminate called after throwing an instance of 'std::runtime_error'
  what():  false CHECK FAILED at ../c10/core/Backend.h"><pre class="notranslate"><code class="notranslate">type id is VariableTensorId
terminate called after throwing an instance of 'std::runtime_error'
  what():  false CHECK FAILED at ../c10/core/Backend.h
</code></pre></div>
<p dir="auto">Seems we need AutoGradMode guard and AutoNonVariableTypeMode guard while<br>
torch::jit::load().</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">In pytorch 1.3, when doing inference with resnet34 on CPU with variable input shapes, much more memory is used compared to pytorch 1.1 (both CPU-only builds on one core): 6 GB for pytorch 1.3 vs. ~0.5 GB for pytorch 1.1</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">Run the following script <a href="https://gist.github.com/lopuhin/0d100ef7df01fdfc91d9685f6e01ff64">https://gist.github.com/lopuhin/0d100ef7df01fdfc91d9685f6e01ff64</a> - it performs inference with resnet34 on images with fixed width and variable height, and reports speed and memory growth over the course of the benchmark.</p>
<p dir="auto">Running under pytorch 1.1:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python3 pytorch_high_mem.py --n 500
torch 1.1.0
heights: mean=1004, p50=278 p95=5100 max=7680
n=100 memory growth (kb): 477,952
n=200 memory growth (kb): 503,948
n=300 memory growth (kb): 503,948
n=400 memory growth (kb): 518,652
time: mean=0.924 s, p50=0.271 s, p95=4.626 s
memory (kb): 174,552 initial, 518,652 growth"><pre class="notranslate"><code class="notranslate">$ python3 pytorch_high_mem.py --n 500
torch 1.1.0
heights: mean=1004, p50=278 p95=5100 max=7680
n=100 memory growth (kb): 477,952
n=200 memory growth (kb): 503,948
n=300 memory growth (kb): 503,948
n=400 memory growth (kb): 518,652
time: mean=0.924 s, p50=0.271 s, p95=4.626 s
memory (kb): 174,552 initial, 518,652 growth
</code></pre></div>
<p dir="auto">Running under pytorch 1.3:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ python3 pytorch_high_mem.py --n 500
torch 1.3.0+cpu
heights: mean=1004, p50=278 p95=5100 max=7680
n=100 memory growth (kb): 2,624,296
n=200 memory growth (kb): 4,480,012
n=300 memory growth (kb): 5,579,568
n=400 memory growth (kb): 5,600,888
time: mean=0.676 s, p50=0.196 s, p95=3.825 s
memory (kb): 187,840 initial, 6,200,664 growth"><pre class="notranslate"><code class="notranslate">$ python3 pytorch_high_mem.py --n 500
torch 1.3.0+cpu
heights: mean=1004, p50=278 p95=5100 max=7680
n=100 memory growth (kb): 2,624,296
n=200 memory growth (kb): 4,480,012
n=300 memory growth (kb): 5,579,568
n=400 memory growth (kb): 5,600,888
time: mean=0.676 s, p50=0.196 s, p95=3.825 s
memory (kb): 187,840 initial, 6,200,664 growth
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Expected behavior is low memory usage as in pytorch 1.1. Alternatively, a way to control caching (e.g. something which disables caching or something like <code class="notranslate">torch.cuda.clear_caches()</code> but for CPU) - as I understand, high memory usage happens because allocations are cached, which makes sense for fixed shapes, but does not work well for variable shapes. Binning shapes is possible as a work-around but has a noticeable performance penalty and memory usage is still higher.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Environment under pytorch 1.1 (via <code class="notranslate">collect_env.py</code> script):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Debian GNU/Linux 9 (stretch)
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torchvision==0.3.0
[conda] Could not collect"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Debian GNU/Linux 9 (stretch)
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torchvision==0.3.0
[conda] Could not collect
</code></pre></div>
<p dir="auto">pytorch installed with</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -U --no-cache-dir cython wheel pip http://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl http://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whl"><pre class="notranslate"><code class="notranslate">pip install -U --no-cache-dir cython wheel pip http://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl http://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whl
</code></pre></div>
<p dir="auto">Environment under pytorch 1.3:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PyTorch version: 1.3.0+cpu
Is debug build: No
CUDA used to build PyTorch: None

OS: Debian GNU/Linux 9 (stretch)
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.3.0+cpu
[pip3] torchvision==0.4.1+cpu
[conda] Could not collect"><pre class="notranslate"><code class="notranslate">PyTorch version: 1.3.0+cpu
Is debug build: No
CUDA used to build PyTorch: None

OS: Debian GNU/Linux 9 (stretch)
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] torch==1.3.0+cpu
[pip3] torchvision==0.4.1+cpu
[conda] Could not collect
</code></pre></div>
<p dir="auto">pytorch installed with</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install torch==1.3.0+cpu torchvision==0.4.1+cpu -f https://download.pytorch.org/whl/torch_stable.html"><pre class="notranslate"><code class="notranslate">pip install torch==1.3.0+cpu torchvision==0.4.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
</code></pre></div>
<h2 dir="auto">Additional context</h2>
<p dir="auto">This may be similar to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="453596809" data-permission-text="Title is private" data-url="https://github.com/oneapi-src/oneDNN/issues/489" data-hovercard-type="issue" data-hovercard-url="/oneapi-src/oneDNN/issues/489/hovercard" href="https://github.com/oneapi-src/oneDNN/issues/489">oneapi-src/oneDNN#489</a> but here mkldnn is not used explicitly.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/VitalyFedyunin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/VitalyFedyunin">@VitalyFedyunin</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gujinghui/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gujinghui">@gujinghui</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/PenghuiCheng/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/PenghuiCheng">@PenghuiCheng</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/XiaobingSuper/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/XiaobingSuper">@XiaobingSuper</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jianyuh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jianyuh">@jianyuh</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a></p>