<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>

<p dir="auto">I am using both PyTorch 1.5 (stable) and nightly build (1.6.0.dev20200611).<br>
Sometimes I need to run code in v1.5 and use files saved by <code class="notranslate">torch.save</code> from the nightly build.<br>
However, PyTorch 1.5 fails to <code class="notranslate">torch.load</code> such files.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ul dir="auto">
<li>PyTorch nightly</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
a = torch.randn(1)
torch.save(a, 'a.pt')"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">randn</span>(<span class="pl-c1">1</span>)
<span class="pl-s1">torch</span>.<span class="pl-en">save</span>(<span class="pl-s1">a</span>, <span class="pl-s">'a.pt'</span>)</pre></div>
<ul dir="auto">
<li>PyTorch 1.5</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
torch.load('a.pt')"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-s1">torch</span>.<span class="pl-en">load</span>(<span class="pl-s">'a.pt'</span>)</pre></div>
<p dir="auto">Then it shows the following error message</p>
<p dir="auto">Traceback (most recent call last):<br>
File "", line 1, in <br>
File "/home/seungjun/local/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/serialization.py", line 586, in load<br>
with _open_zipfile_reader(f) as opened_zipfile:<br>
File "/home/seungjun/local/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/serialization.py", line 246, in <strong>init</strong><br>
super(_open_zipfile_reader, self).<strong>init</strong>(torch.<em>C.PyTorchFileReader(name_or_buffer))<br>
RuntimeError: version</em> &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1587428266983/work/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 4, but the maximum supported version for reading is 3. Your PyTorch installation may be too old.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">I expect PyTorch 1.5 to load files without error.<br>
Current behavior could be due to the update in PyTorchFileReader but I assume 1.5 and 1.6 should be compatible as their version number is close to each other.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Loading:</p>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.0): 1.5.0</li>
<li>OS (e.g., Linux): Ubuntu 18.04.4 LTS</li>
<li>How you installed PyTorch (<code class="notranslate">conda</code>, <code class="notranslate">pip</code>, source): conda</li>
<li>Build command you used (if compiling from source):<br>
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch</li>
<li>Python version: 3.7</li>
<li>CUDA/cuDNN version: 10.2.89, 7.6.5</li>
</ul>
<p dir="auto">Saving:</p>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.0): 1.6.0.dev20200611 for saving</li>
<li>OS (e.g., Linux): Ubuntu 18.04.4 LTS</li>
<li>How you installed PyTorch (<code class="notranslate">conda</code>, <code class="notranslate">pip</code>, source): conda</li>
<li>Build command you used (if compiling from source):<br>
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch-nightly</li>
<li>Python version: 3.8</li>
<li>CUDA/cuDNN version: 10.2.89, 7.6.5</li>
</ul>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">This issus is same as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="643650470" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/40425" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/40425/hovercard" href="https://github.com/pytorch/pytorch/issues/40425">#40425</a>, but  40425 can not slove my question, so i reopen it again ,and add some NOTE.</p>
<p dir="auto">I am using cusolverDn in c++ and import it to python with pybind11.<br>
After use cusolverDnCreate() and cusolverDnDestroy() with torch,Segment Fault appear when program destruction.</p>
<p dir="auto"><strong>NOTE:</strong></p>
<h3 dir="auto"><strong>1: When the demo run with Torch1.2,it run success, the SegFault disappear, does it mean Torch 1.5 has some bug?</strong></h3>
<h3 dir="auto"><strong>2: This SegFault only appear with "thread_local", Does it provide some information for this bug?</strong></h3>
<h3 dir="auto"><strong>3: Please see test.py below, if we "import torch" after "import example", SegFault disappear.</strong></h3>
<h3 dir="auto"><strong>4: Torch have not link libcusolver.so, so we just link it once.</strong></h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> say that upgarde cuda may fix this.But i can not upgrade in production environment。</p>
<p dir="auto">cc please <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yf225/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yf225">@yf225</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/glaringlee/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/glaringlee">@glaringlee</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mcarilli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mcarilli">@mcarilli</a></p>

<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">1.paste the c++ code, name bind_cudevice.cc<br>
2.compile bind_cudevice.cc with my g++ compile code<br>
3.paste my python code named test.py<br>
4. just run test.py and see</p>
<p dir="auto">bind_cudevice.cc</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#include &lt;iostream&gt;

#include &lt;cublas_v2.h&gt;
#include &lt;cuda.h&gt;
#include &lt;cuda_runtime_api.h&gt;

#include &lt;cusolverDn.h&gt;
#include &lt;cusolverSp.h&gt;
#include &lt;cusolverRf.h&gt;

#include &lt;pybind11/pybind11.h&gt;

class CuDevice {

public:
     CuDevice() {
        initialized_ = true;

        cudaSetDevice(0);

        cusolverDnCreate(&amp;cusolverdn_handle_);
        cusolverDnSetStream(cusolverdn_handle_, cudaStreamPerThread);
    }

    ~CuDevice() {
        std::cout &lt;&lt; &quot;in ~CuDevice&quot; &lt;&lt; std::endl;
        if (cusolverdn_handle_) {
            std::cout &lt;&lt; &quot;in ~CuDevice and cusolverdn_handle_&quot; &lt;&lt; std::endl;
            cusolverDnDestroy(cusolverdn_handle_);
            std::cout &lt;&lt; &quot;Destory success&quot; &lt;&lt; std::endl;
       }
    }

private:
    bool initialized_;
    cusolverDnHandle_t cusolverdn_handle_;

};

int init_cudevice(){
    thread_local CuDevice a = CuDevice();
    std::cout &lt;&lt; &quot;hello world&quot; &lt;&lt; std::endl;
    return 0;
}

PYBIND11_MODULE(example, m) {
    m.def(&quot;init_cudevide&quot;,
    &amp;init_cudevice,
    &quot;select gpu id&quot;);
}"><pre class="notranslate"><code class="notranslate">#include &lt;iostream&gt;

#include &lt;cublas_v2.h&gt;
#include &lt;cuda.h&gt;
#include &lt;cuda_runtime_api.h&gt;

#include &lt;cusolverDn.h&gt;
#include &lt;cusolverSp.h&gt;
#include &lt;cusolverRf.h&gt;

#include &lt;pybind11/pybind11.h&gt;

class CuDevice {

public:
     CuDevice() {
        initialized_ = true;

        cudaSetDevice(0);

        cusolverDnCreate(&amp;cusolverdn_handle_);
        cusolverDnSetStream(cusolverdn_handle_, cudaStreamPerThread);
    }

    ~CuDevice() {
        std::cout &lt;&lt; "in ~CuDevice" &lt;&lt; std::endl;
        if (cusolverdn_handle_) {
            std::cout &lt;&lt; "in ~CuDevice and cusolverdn_handle_" &lt;&lt; std::endl;
            cusolverDnDestroy(cusolverdn_handle_);
            std::cout &lt;&lt; "Destory success" &lt;&lt; std::endl;
       }
    }

private:
    bool initialized_;
    cusolverDnHandle_t cusolverdn_handle_;

};

int init_cudevice(){
    thread_local CuDevice a = CuDevice();
    std::cout &lt;&lt; "hello world" &lt;&lt; std::endl;
    return 0;
}

PYBIND11_MODULE(example, m) {
    m.def("init_cudevide",
    &amp;init_cudevice,
    "select gpu id");
}
</code></pre></div>
<p dir="auto">g++ compile code</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
g++ -O3 -Wall -shared -std=c++11 -fPIC -I/usr/include/python3.6m -I./third-party/pybind11-2.2.4/include -I/usr/local/cuda/include -Wl,-rpath=/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -lcudart -lcusolver `python3 -m pybind11 --includes` binding_cudevice.cc -o example`python3-config --extension-suffix`
"><pre class="notranslate"><code class="notranslate">
g++ -O3 -Wall -shared -std=c++11 -fPIC -I/usr/include/python3.6m -I./third-party/pybind11-2.2.4/include -I/usr/local/cuda/include -Wl,-rpath=/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -lcudart -lcusolver `python3 -m pybind11 --includes` binding_cudevice.cc -o example`python3-config --extension-suffix`

</code></pre></div>
<p dir="auto">test.py</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import example
example.init_cudevide()"><pre class="notranslate"><code class="notranslate">import torch
import example
example.init_cudevide()
</code></pre></div>
<h2 dir="auto">Core Dump Info</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(gdb) bt
#0  0x00007f78b6a391de in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#1  0x00007f78b6a5b4ca in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#2  0x00007f78b5db6e49 in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#3  0x00007f78b5b854c1 in cusolverDnDestroy () from /usr/local/cuda/lib64/libcusolver.so.10.0
#4  0x00007f78be143952 in kaldi::CuDevice::~CuDevice (this=0x2f18a38, __in_chrg=&lt;optimized out&gt;) at cu-device.cc:683
#5  0x00007f79880d7ad1 in (anonymous namespace)::run (p=&lt;optimized out&gt;) at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/atexit_thread.cc:75
#6  0x00007f79b752cc99 in __run_exit_handlers () from /usr/lib64/libc.so.6
#7  0x00007f79b752cce7 in exit () from /usr/lib64/libc.so.6
#8  0x00007f79b751550c in __libc_start_main () from /usr/lib64/libc.so.6
#9  0x0000000000400c20 in _start ()"><pre class="notranslate"><code class="notranslate">(gdb) bt
#0  0x00007f78b6a391de in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#1  0x00007f78b6a5b4ca in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#2  0x00007f78b5db6e49 in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#3  0x00007f78b5b854c1 in cusolverDnDestroy () from /usr/local/cuda/lib64/libcusolver.so.10.0
#4  0x00007f78be143952 in kaldi::CuDevice::~CuDevice (this=0x2f18a38, __in_chrg=&lt;optimized out&gt;) at cu-device.cc:683
#5  0x00007f79880d7ad1 in (anonymous namespace)::run (p=&lt;optimized out&gt;) at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/atexit_thread.cc:75
#6  0x00007f79b752cc99 in __run_exit_handlers () from /usr/lib64/libc.so.6
#7  0x00007f79b752cce7 in exit () from /usr/lib64/libc.so.6
#8  0x00007f79b751550c in __libc_start_main () from /usr/lib64/libc.so.6
#9  0x0000000000400c20 in _start ()
</code></pre></div>

<h2 dir="auto">Expected behavior</h2>
<p dir="auto">No error happen</p>

<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.5):1.5</li>
<li>OS (e.g., Linux):Centos7</li>
<li>How you installed PyTorch (<code class="notranslate">conda</code>, <code class="notranslate">pip</code>, source):source</li>
<li>Build command you used (if compiling from source):</li>
<li>Python version:3.6.8</li>
<li>CUDA/cuDNN version:10.0</li>
<li>GPU models and configuration:K40</li>
<li>Any other relevant information:</li>
</ul>
<h2 dir="auto">Additional context</h2>

<p dir="auto">cuSolver is base cublas and cusparse, reference about cuSolver：<br>
<a href="https://docs.nvidia.com/cuda/archive/10.0/cusolver/index.html#cuSolverDNhandle" rel="nofollow">https://docs.nvidia.com/cuda/archive/10.0/cusolver/index.html#cuSolverDNhandle</a></p>