I got segmentation fault when running mmdetection program on a HPC (CentOS 6.9). Then I used gdb and got the following message:
GNU gdb (GDB) 7.6.1
Copyright © 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-unknown-linux-gnu”.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/…
Reading symbols from /usr/local/python/3.5.1/bin/python3.5…done.
Starting program: /project/RDS-FEI-cvpr19-RW/mmdetection/pytorch4/bin/python tools/train.py configs/faster_rcnn_r50_fpn_1x_ucf24.py --gpus 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib64/libthread_db.so.1”.
[New Thread 0x2aab74179700 (LWP 175221)]
[New Thread 0x2aab7437a700 (LWP 175222)]
2018-12-03 23:01:21,200 - INFO - Distributed training: False
2018-12-03 23:01:23,187 - INFO - load model from: modelzoo://resnet50
2018-12-03 23:01:26,339 - INFO - load checkpoint from pretrain_stage2.pth
2018-12-03 23:01:26,716 - INFO - Start running, host: rusu5516@hpc222, work_dir: /project/RDS-FEI-cvpr19-RW/mmdetection/work_dirs/faster_rcnn_r50_fpn_1x_full_data_stage2_scratch_2x
2018-12-03 23:01:26,717 - INFO - workflow: [(‘train’, 1)], max: 12 epochs
[New Thread 0x2aab64009700 (LWP 175224)]
[New Thread 0x2aabadeeb700 (LWP 175225)]
Program received signal SIGSEGV, Segmentation fault.
0x00002aab51f395c0 in construct<_object*, _object*> (__p=0xb, this=0x1365578)
at /usr/local/gcc/4.9.0/include/c++/4.9.0/ext/new_allocator.h:120
120		{ ::new((void *)__p) _Up(std::forward<_Args>(__args)…); }
(gdb) quit
A debugging session is active.
Inferior 1 [process 175019] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]
I can successfully run the same program on my local machine (ubuntu 16.04). Is there anyone knowing what is going on here?