<p dir="auto">Comments to 4518 have multitude of suggestions, but no clear resolution.</p>
<p dir="auto">I am on Win10 no CUDA</p>
<p dir="auto">Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32<br>
Type "help", "copyright", "credits" or "license" for more information.</p>
<p dir="auto">import torch<br>
Traceback (most recent call last):<br>
File "", line 1, in<br>
File "C:\Users\dae\AppData\Local\Programs\Python\Python36\lib\site-packages\torch_init_.py", line 78, in<br>
from torch._C import *<br>
ImportError: DLL load failed: The specified module could not be found.</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Pytorch-1.0-Cuda-10.0 core dumps on Ubuntu Linux on some cpu types</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">Spin up an Ubuntu with GPU instance like EC2 p2 or g3</p>
<p dir="auto">install Cuda-10, cudnn-7.4.1, NCCL-2.3.7<br>
install anaconda3-5.1.0<br>
conda create -n pytorch_p27 python=2.7<br>
conda install pytorch=1.0.0=py2.7_cuda10.0.130_cudnn7.4.1_1 torchvision=0.2.1 cuda100=1.0 -c pytorch</p>
<p dir="auto">example used to replicate:<br>
<a href="https://github.com/pytorch/examples/tree/master/mnist">https://github.com/pytorch/examples/tree/master/mnist</a><br>
branch 0.4</p>
<p dir="auto">cd mnist<br>
python main.py</p>
<p dir="auto">Error:<br>
Illegal instruction (core dumped)</p>
<p dir="auto">gdb output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Thread 1 &quot;python&quot; received signal SIGILL, Illegal instruction.
0x00007fffc0798eab in mkldnn::impl::scales_t::set(int, int, float const*) () from /usr/local/lib/libmkldnn.so.0
(gdb) bt
#0  0x00007fffde7a4eab in mkldnn::impl::scales_t::set(int, int, float const*) () from /usr/local/lib/libmkldnn.so.0
#1  0x00007fffde7a1b42 in mkldnn_primitive_desc_create_v2 () from /usr/local/lib/libmkldnn.so.0
#2  0x00007fff12114b40 in mkldnn::convolution_forward::primitive_desc::primitive_desc(mkldnn::convolution_forward::desc const&amp;, mkldnn::engine const&amp;) ()
   from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#3  0x00007fff1210df05 in at::native::mkldnn_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#4  0x00007fff1227dec0 in at::TypeDefault::mkldnn_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#5  0x00007fffcee92d18 in torch::autograd::VariableType::mkldnn_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#6  0x00007fff11fa4f92 in at::native::_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#7  0x00007fff122783ae in at::TypeDefault::_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#8  0x00007fffcef1a7da in torch::autograd::VariableType::_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool) const ()
   from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#9  0x00007fff11f9f7d9 in at::native::convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#10 0x00007fff12278287 in at::TypeDefault::convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#11 0x00007fffcef1f464 in torch::autograd::VariableType::convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#12 0x00007fff11f9f4aa in at::native::conv2d(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#13 0x00007fff122787c0 in at::TypeDefault::conv2d(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#14 0x00007fffcefd8cf6 in torch::autograd::VariableType::conv2d(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#15 0x00007fffe811fd1b in torch::autograd::THPVariable_conv2d(_object*, _object*, _object*) ()
   from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch_python.so
#16 0x00007ffff7adbcc4 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#17 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#18 0x00007ffff7a66377 in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#19 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#20 0x00007ffff7ad64be in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#21 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#22 0x00007ffff7a6628a in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#23 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#24 0x00007ffff7a5063d in instancemethod_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#25 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#26 0x00007ffff7a9a8a4 in slot_tp_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#27 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#28 0x00007ffff7ad7b69 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#29 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#30 0x00007ffff7a66377 in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#31 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#32 0x00007ffff7ad64be in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#33 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#34 0x00007ffff7a6628a in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#35 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#36 0x00007ffff7a5063d in instancemethod_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#37 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#38 0x00007ffff7a9a8a4 in slot_tp_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#39 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#40 0x00007ffff7ad7b69 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#41 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#42 0x00007ffff7ada9b8 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#43 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#44 0x00007ffff7ada9b8 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#45 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#46 0x00007ffff7add70a in PyEval_EvalCode () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#47 0x00007ffff7af69cd in run_mod () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#48 0x00007ffff7af7b48 in PyRun_FileExFlags () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#49 0x00007ffff7af8d68 in PyRun_SimpleFileExFlags () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#50 0x00007ffff7b0af8c in Py_Main () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#51 0x00007ffff6d22830 in __libc_start_main (main=0x555555554890 &lt;main&gt;, argc=2, argv=0x7fffffffde98, init=&lt;optimized out&gt;, fini=&lt;optimized out&gt;, 
    rtld_fini=&lt;optimized out&gt;, stack_end=0x7fffffffde88) at ../csu/libc-start.c:291
#52 0x00005555555548bf in _start ()
(gdb) n
Single stepping until exit from function _ZN6mkldnn4impl8scales_t3setEiiPKf,
which has no line number information.

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists."><pre class="notranslate"><code class="notranslate">Thread 1 "python" received signal SIGILL, Illegal instruction.
0x00007fffc0798eab in mkldnn::impl::scales_t::set(int, int, float const*) () from /usr/local/lib/libmkldnn.so.0
(gdb) bt
#0  0x00007fffde7a4eab in mkldnn::impl::scales_t::set(int, int, float const*) () from /usr/local/lib/libmkldnn.so.0
#1  0x00007fffde7a1b42 in mkldnn_primitive_desc_create_v2 () from /usr/local/lib/libmkldnn.so.0
#2  0x00007fff12114b40 in mkldnn::convolution_forward::primitive_desc::primitive_desc(mkldnn::convolution_forward::desc const&amp;, mkldnn::engine const&amp;) ()
   from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#3  0x00007fff1210df05 in at::native::mkldnn_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#4  0x00007fff1227dec0 in at::TypeDefault::mkldnn_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#5  0x00007fffcee92d18 in torch::autograd::VariableType::mkldnn_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#6  0x00007fff11fa4f92 in at::native::_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#7  0x00007fff122783ae in at::TypeDefault::_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#8  0x00007fffcef1a7da in torch::autograd::VariableType::_convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool) const ()
   from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#9  0x00007fff11f9f7d9 in at::native::convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#10 0x00007fff12278287 in at::TypeDefault::convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#11 0x00007fffcef1f464 in torch::autograd::VariableType::convolution(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, bool, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#12 0x00007fff11f9f4aa in at::native::conv2d(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#13 0x00007fff122787c0 in at::TypeDefault::conv2d(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libcaffe2.so
#14 0x00007fffcefd8cf6 in torch::autograd::VariableType::conv2d(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long) const () from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch.so.1
#15 0x00007fffe811fd1b in torch::autograd::THPVariable_conv2d(_object*, _object*, _object*) ()
   from /home/ubuntu/anaconda3/envs/pytorch_p27/lib/python2.7/site-packages/torch/lib/libtorch_python.so
#16 0x00007ffff7adbcc4 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#17 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#18 0x00007ffff7a66377 in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#19 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#20 0x00007ffff7ad64be in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#21 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#22 0x00007ffff7a6628a in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#23 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#24 0x00007ffff7a5063d in instancemethod_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#25 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#26 0x00007ffff7a9a8a4 in slot_tp_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#27 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#28 0x00007ffff7ad7b69 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#29 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#30 0x00007ffff7a66377 in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#31 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#32 0x00007ffff7ad64be in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#33 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#34 0x00007ffff7a6628a in function_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#35 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#36 0x00007ffff7a5063d in instancemethod_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#37 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#38 0x00007ffff7a9a8a4 in slot_tp_call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#39 0x00007ffff7a417a3 in PyObject_Call () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#40 0x00007ffff7ad7b69 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#41 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#42 0x00007ffff7ada9b8 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#43 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#44 0x00007ffff7ada9b8 in PyEval_EvalFrameEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#45 0x00007ffff7add4e9 in PyEval_EvalCodeEx () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#46 0x00007ffff7add70a in PyEval_EvalCode () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#47 0x00007ffff7af69cd in run_mod () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#48 0x00007ffff7af7b48 in PyRun_FileExFlags () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#49 0x00007ffff7af8d68 in PyRun_SimpleFileExFlags () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#50 0x00007ffff7b0af8c in Py_Main () from /home/ubuntu/anaconda3/envs/pytorch_p27/bin/../lib/libpython2.7.so.1.0
#51 0x00007ffff6d22830 in __libc_start_main (main=0x555555554890 &lt;main&gt;, argc=2, argv=0x7fffffffde98, init=&lt;optimized out&gt;, fini=&lt;optimized out&gt;, 
    rtld_fini=&lt;optimized out&gt;, stack_end=0x7fffffffde88) at ../csu/libc-start.c:291
#52 0x00005555555548bf in _start ()
(gdb) n
Single stepping until exit from function _ZN6mkldnn4impl8scales_t3setEiiPKf,
which has no line number information.

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.
</code></pre></div>
<p dir="auto">Program terminated with signal SIGILL, Illegal instruction.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Start training on MNIST dataset</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Please copy and paste the output from our<br>
<a href="https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py" rel="nofollow">environment collection script</a><br>
(or fill out the checklist below manually).</p>
<p dir="auto">You can get the script and run it with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py"><pre class="notranslate"><code class="notranslate">wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.0.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 2.7
Is CUDA available: No
CUDA runtime version: 10.0.130
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/local/cuda-10.0/lib64/libcudnn.so.7.4.1
/usr/local/cuda-10.0/lib64/libcudnn_static.a
/usr/local/cuda-8.0/lib64/libcudnn.so.6.0.21
/usr/local/cuda-8.0/lib64/libcudnn_static.a
/usr/local/cuda-9.0/lib64/libcudnn.so.7.3.1
/usr/local/cuda-9.0/lib64/libcudnn_static.a
/usr/local/cuda-9.2/lib64/libcudnn.so.7.3.1
/usr/local/cuda-9.2/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] numpy (1.14.5)
[pip] numpydoc (0.8.0)
[pip] torch (1.0.0)
[pip] torchvision (0.2.1)
[conda] blas                      1.0                         mkl  
[conda] cuda100                   1.0                           0    pytorch
[conda] mkl                       2018.0.2                      1  
[conda] mkl-service               1.1.2            py27hb2d42c5_4  
[conda] mkl_fft                   1.0.1            py27h3010b51_0  
[conda] mkl_random                1.0.1            py27h629b387_0  
[conda] pytorch                   1.0.0           py2.7_cuda10.0.130_cudnn7.4.1_1  [cuda100]  pytorch
[conda] torchvision               0.2.1                      py_2    pytorch
"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.0.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 2.7
Is CUDA available: No
CUDA runtime version: 10.0.130
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/local/cuda-10.0/lib64/libcudnn.so.7.4.1
/usr/local/cuda-10.0/lib64/libcudnn_static.a
/usr/local/cuda-8.0/lib64/libcudnn.so.6.0.21
/usr/local/cuda-8.0/lib64/libcudnn_static.a
/usr/local/cuda-9.0/lib64/libcudnn.so.7.3.1
/usr/local/cuda-9.0/lib64/libcudnn_static.a
/usr/local/cuda-9.2/lib64/libcudnn.so.7.3.1
/usr/local/cuda-9.2/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] numpy (1.14.5)
[pip] numpydoc (0.8.0)
[pip] torch (1.0.0)
[pip] torchvision (0.2.1)
[conda] blas                      1.0                         mkl  
[conda] cuda100                   1.0                           0    pytorch
[conda] mkl                       2018.0.2                      1  
[conda] mkl-service               1.1.2            py27hb2d42c5_4  
[conda] mkl_fft                   1.0.1            py27h3010b51_0  
[conda] mkl_random                1.0.1            py27h629b387_0  
[conda] pytorch                   1.0.0           py2.7_cuda10.0.130_cudnn7.4.1_1  [cuda100]  pytorch
[conda] torchvision               0.2.1                      py_2    pytorch

</code></pre></div>
<h2 dir="auto">Additional context</h2>
<p dir="auto">CPU types where the problem occurs:<br>
EC2 instance type - i3.16xlarge</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                64
On-line CPU(s) list:   0-63
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               2199.914
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4600.08
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt ida"><pre class="notranslate"><code class="notranslate">Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                64
On-line CPU(s) list:   0-63
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               2199.914
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4600.08
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt ida
</code></pre></div>
<p dir="auto">EC2 instance type - c4.8xlarge</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                36
On-line CPU(s) list:   0-35
Thread(s) per core:    2
Core(s) per socket:    9
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
Stepping:              2
CPU MHz:               1201.234
CPU max MHz:           3500.0000
CPU min MHz:           1200.0000
BogoMIPS:              5800.10
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-8,18-26
NUMA node1 CPU(s):     9-17,27-35
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt ida"><pre class="notranslate"><code class="notranslate">Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                36
On-line CPU(s) list:   0-35
Thread(s) per core:    2
Core(s) per socket:    9
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
Stepping:              2
CPU MHz:               1201.234
CPU max MHz:           3500.0000
CPU min MHz:           1200.0000
BogoMIPS:              5800.10
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-8,18-26
NUMA node1 CPU(s):     9-17,27-35
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt ida
</code></pre></div>