Potential Bug, I am desperate for some help with this.

  

      github.com/pytorch/pytorch
  

  
    
  
	  
  

  
    
      Libtorch: Segmentation fault when running torch::jit::load 16
    

    
      
        opened 
        
          
        
        Dec 16, 2020
      
      


      
        
          
          jarledo
        
      
    

    
        oncall: jit
        weeks
    
  


  
    ## 🐛 Bug

When trying to load some of our models exported from Python using `t…orch.jit.trace` in a C++ program using `torch::jit::load("/path/to/model")` it causes a segmentation fault. The segfault seems to happen in `torch::jit::SourceImporterImpl::attributeAssignmentSpecialHandlingHack`

## To Reproduce

I'm unable to generate a minimal example that causes the segmentation fault. It only happens with some of our models when linked to our custom application. The unit tests that load the models work just fine, so perhaps there is some access to memory location that has not been allocated that only triggers during very specific linking with our application?

A stack trace from gdb:
```
Thread 1 "myprogram" received signal SIGSEGV, Segmentation fault.
0x0000555555742bf9 in std::enable_if<std::__detail::__is_contiguous_iter<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::value, std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::__string_view>::type std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_str<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >() const ()
#0  0x0000555555742bf9 in std::enable_if<std::__detail::__is_contiguous_iter<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::value, std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::__string_view>::type std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_str<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >() const ()
#1  0x0000555555741e06 in std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::compare(std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const ()
#2  0x0000555555741014 in bool std::__cxx11::operator==<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >(std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ()
#3  0x00005555557402f8 in std::__cxx11::regex_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, char, std::__cxx11::regex_traits<char> >::operator==(std::__cxx11::regex_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, char, std::__cxx11::regex_traits<char> > const&) const ()
#4  0x00007fffe96d504f in std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > std::regex_replace<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::regex_traits<char>, char>(std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_regex<char, std::__cxx11::regex_traits<char> > const&, char const*, std::regex_constants::match_flag_type) ()
   from /opt/libtorch/lib/libtorch_cpu.so
#5  0x00007fffe97f7349 in torch::jit::SourceImporterImpl::attributeAssignmentSpecialHandlingHack(c10::QualifiedName const&, torch::jit::Assign const&) ()
   from /opt/libtorch/lib/libtorch_cpu.so
#6  0x00007fffe97f9e66 in torch::jit::SourceImporterImpl::importClass(c10::QualifiedName const&, torch::jit::ClassDef const&, bool) () from /opt/libtorch/lib/libtorch_cpu.so
#7  0x00007fffe97fa9fb in torch::jit::SourceImporterImpl::importNamedType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, torch::jit::ClassDef const&) () from /opt/libtorch/lib/libtorch_cpu.so
#8  0x00007fffe97fae05 in torch::jit::SourceImporterImpl::findNamedType(c10::QualifiedName const&) () from /opt/libtorch/lib/libtorch_cpu.so
#9  0x00007fffe97faeea in torch::jit::SourceImporterImpl::resolveType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, torch::jit::SourceRange const&)
    () from /opt/libtorch/lib/libtorch_cpu.so
#10 0x00007fffe9568821 in torch::jit::ScriptTypeParser::parseTypeFromExpr(torch::jit::Expr const&) const () from /opt/libtorch/lib/libtorch_cpu.so
#11 0x00007fffe9568ae9 in torch::jit::ScriptTypeParser::parseType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /opt/libtorch/lib/libtorch_cpu.so
#12 0x00007fffe97ecfbb in torch::jit::SourceImporter::loadType(c10::QualifiedName const&) const () from /opt/libtorch/lib/libtorch_cpu.so
#13 0x00007fffe97d912b in std::_Function_handler<c10::StrongTypePtr (c10::QualifiedName const&), torch::jit::(anonymous namespace)::ScriptModuleDeserializer::readArchive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda(c10::QualifiedName const&)#1}>::_M_invoke(std::_Any_data const&, c10::QualifiedName const&) ()
   from /opt/libtorch/lib/libtorch_cpu.so
#14 0x00007fffe982e16b in torch::jit::Unpickler::readGlobal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /opt/libtorch/lib/libtorch_cpu.so
#15 0x00007fffe9832c02 in torch::jit::Unpickler::readInstruction() ()
   from /opt/libtorch/lib/libtorch_cpu.so
#16 0x00007fffe9835840 in torch::jit::Unpickler::run() ()
   from /opt/libtorch/lib/libtorch_cpu.so
#17 0x00007fffe9835df1 in torch::jit::Unpickler::parse_ivalue() ()
   from /opt/libtorch/lib/libtorch_cpu.so
#18 0x00007fffe97d9cc2 in torch::jit::readArchiveAndTensors(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<std::function<c10::StrongTypePtr (c10::QualifiedName const&)> >, c10::optional<std::function<c10::intrusive_ptr<c10::ivalue::Object, c10::detail::intrusive_target_default_null_type<c10::ivalue::Object> > (c10::StrongTypePtr, c10::IValue)> >, c10::optional<c10::Device>, caffe2::serialize::PyTorchStreamReader&)
    () from /opt/libtorch/lib/libtorch_cpu.so
#19 0x00007fffe97d9fdd in torch::jit::(anonymous namespace)::ScriptModuleDeserializer::readArchive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .constprop.789] () from /opt/libtorch/lib/libtorch_cpu.so
#20 0x00007fffe97dc905 in torch::jit::(anonymous namespace)::ScriptModuleDeserializer::deserialize(c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) [clone .constprop.788] ()
   from /opt/libtorch/lib/libtorch_cpu.so
#21 0x00007fffe97dced9 in torch::jit::load(std::unique_ptr<caffe2::serialize::ReadAdapterInterface, std::default_delete<caffe2::serialize::ReadAdapterInterface> >, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) () from /opt/libtorch/lib/libtorch_cpu.so
#22 0x00007fffe97df755 in torch::jit::load(std::istream&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) ()
   from /opt/libtorch/lib/libtorch_cpu.so
```

## Expected behavior

The program should never cause segmentation fault.

## Environment

 - PyTorch Version (e.g., 1.0): Libtorch v1.7.1, both cpu and Cuda versions produce same error
 - OS (e.g., Linux): Ubuntu 20.04 and Ubuntu 18.04
 - How you installed PyTorch (`conda`, `pip`, source): Binary download of libtorch
 - CUDA/cuDNN version: CPU and CUDA 11.0 with cuDNN 8.0.5
 - GPU models and configuration: CPU and RTX 2070
 - Any other relevant information: Tested on several computers in several different docker environments

## Additional context

Would it be helpful if I attempt to build libtorch from source with debug symbols to produce a better stack trace in gdb and inspect local variables?


cc @gmagogsfm
  

  

  
    
    
  

  


I have run into the exact issue linked above.
My reproducable example:

Redhat 7.9
Matlab 2020a
libtorch 1.7.1

I create a .so which has a single function inside, all this function does is call torch::jit::load. Matlab loads in the .so through calllib(), when the .so function is called and torch::jit::load is invoked a segmentation fault will result and everything will crash in the exact way that has been linked. It seems that this is an open bug with several other individuals facing this same problem. Could a staff member step in to take a look at this?