While trying to run a program on an RHEL cluster, the code gets stuck at the backward step. The RHEL pytroch is a gpu version. This problem does not occur on my PC which has a cpu version.
Any chance it’s related to: https://github.com/pytorch/pytorch/pull/1243 9? Anyway, how do I go about fixing it?
Below is the gdb backtrace:
(gdb) backtrace
#0  pthread_cond_wait@@GLIBC_2.3.2 () at …/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007fffc7ac3a26 in std::condition_variable::wait(std::unique_lockstd::mutex&) ()
at /home/msarahan/miniconda2/conda-bld/compilers_linux-64_1507259624353/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc+±v3/include/x86_64-conda_cos6-linux-gnu/bits/gthr-default.h:877
#2  0x00007fffe8886573 in torch::autograd::Engine::execute(std::vector<torch::autograd::Edge, std::allocatortorch::autograd::Edge > const&, std::vector<torch::autograd::Variable, std::allocatortorch::autograd::Variable > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocatortorch::autograd::Edge > const&) ()
at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/condition_variable:98
#3  0x00007fffe88b1d7c in torch::autograd::python::PythonEngine::execute(std::vector<torch::autograd::Edge, std::allocatortorch::autograd::Edge > const&, std::vector<torch::autograd::Variable, std::allocatortorch::autograd::Variable > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocatortorch::autograd::Edge > const&) ()
at torch/csrc/autograd/python_engine.cpp:58
#4  0x00007fffe88b29ae in THPEngine_run_backward(THPEngine*, _object*, _object*) ()
at torch/csrc/autograd/python_engine.cpp:166
#5  0x00007ffff7e4ffd4 in _PyCFunction_FastCallDict ()
#6  0x00007ffff7e7df24 in _PyCFunction_FastCallKeywords ()
#7  0x00007ffff7eddbec in call_function ()
#8  0x00007ffff7f02eb1 in _PyEval_EvalFrameDefault ()
#9  0x00007ffff7ed69a6 in _PyEval_EvalCodeWithName ()
#10 0x00007ffff7ed7a11 in fast_function ()
#11 0x00007ffff7eddcc5 in call_function ()
#12 0x00007ffff7f0219a in _PyEval_EvalFrameDefault ()
#13 0x00007ffff7ed69a6 in _PyEval_EvalCodeWithName ()
#14 0x00007ffff7ed7a11 in fast_function ()
#15 0x00007ffff7eddcc5 in call_function ()
#16 0x00007ffff7f02eb1 in _PyEval_EvalFrameDefault ()
#17 0x00007ffff7ed8529 in PyEval_EvalCodeEx ()
#18 0x00007ffff7ed92cc in PyEval_EvalCode ()
#19 0x00007ffff7effa8d in builtin_exec ()
#20 0x00007ffff7e53051 in PyCFunction_Call ()
#21 0x00007ffff7f0742b in _PyEval_EvalFrameDefault ()
#22 0x00007ffff7ed69a6 in _PyEval_EvalCodeWithName ()
#23 0x00007ffff7ed7a11 in fast_function ()
#24 0x00007ffff7eddcc5 in call_function ()
#25 0x00007ffff7f0219a in _PyEval_EvalFrameDefault ()
#26 0x00007ffff7ed77db in fast_function ()
#27 0x00007ffff7eddcc5 in call_function ()
—Type  to continue, or q  to quit—
#28 0x00007ffff7f0219a in _PyEval_EvalFrameDefault ()
#29 0x00007ffff7ed77db in fast_function ()
#30 0x00007ffff7eddcc5 in call_function ()
#31 0x00007ffff7f0219a in _PyEval_EvalFrameDefault ()
#32 0x00007ffff7ed77db in fast_function ()
#33 0x00007ffff7eddcc5 in call_function ()
#34 0x00007ffff7f0219a in _PyEval_EvalFrameDefault ()
#35 0x00007ffff7ed7e4b in _PyFunction_FastCallDict ()
#36 0x00007ffff7e5039f in _PyObject_FastCallDict ()
#37 0x00007ffff7e93650 in _PyObject_CallMethodIdObjArgs ()
#38 0x00007ffff7e46ea0 in PyImport_ImportModuleLevelObject ()
#39 0x00007ffff7f04b2c in _PyEval_EvalFrameDefault ()
#40 0x00007ffff7ed8529 in PyEval_EvalCodeEx ()
#41 0x00007ffff7ed92cc in PyEval_EvalCode ()
#42 0x00007ffff7f55af4 in run_mod ()
#43 0x00007ffff7e1c930 in PyRun_InteractiveOneObjectEx ()
#44 0x00007ffff7e1cae6 in PyRun_InteractiveLoopFlags ()
#45 0x00007ffff7e1cb86 in PyRun_AnyFileExFlags.cold.2769 ()
#46 0x00007ffff7e1eb69 in Py_Main.cold.2794 ()
#47 0x00007ffff7e2171e in main ()
@apaszke - I wonder if you have any ideas since you seem to have taken a look at a similar case before: https://discuss.pytorch.org/t/solved-archlinux-using-variable-backwards-appears-to-hang-program-indefinitely/1675/11 5