I also encounter the same problem,  exactly as what you described. It is very strange the the program some times kill the network. Another strange thing is that the program runs longer in old TITAN X and shorter in 1080Ti. But both will end with segmentation fault(core dumped).
Would it be solved if I downgrade the pytorch to 0.2.0?