yep, the crash occurs so suddenly, since the max_memory_allocated output and nvidia-smi’s output remains constant. Sometimes i can train like 200 epochs while sometimes 10~20 only.
I mainly reference https://github.com/chenyuntc/simple-faster-rcnn-pytorch 3 and https://github.com/pytorch/examples/tree/d8d378c31d2766009db400ac03f41dd837a56c2a/fast_rcnn 3 this two repos. The first one reports that it only takes about 3G GPU memory usage. My program should have smaller memory usage since i’ve only implemented the fast-rcnn part.