I’m getting random jobs aborted when training an LSTM in 2 GPUs, with the following short, cryptic error:
Fatal Python error: deallocating None

Thread 0x00007f260ec2d700 (most recent call first):

Thread 0x00007f260f7c5700 (most recent call first):

Thread 0x00007f260ffc6700 (most recent call first):

Current thread 0x00007f266b197740 (most recent call first):
 File "model.py", line 188 in <module>
Aborted (core dumped)

Line 188 is the training function. Some jobs finish correctly. Some fail like above.
Linux AWS 1044 (Ubuntu 16.04.3)
AWS 8 GPU instance (Tesla K80)
python3.6
pytorch 0.3.0
CUDA 9.0.176
cuDNN 7.0.3
I’m trying to retrieve the kernel core dump to see if there are any clues there. This happens while doing grid-search on hidden size, and seems unrelated to the parameter value.
Not entirely sure how to reproduce it, I’ll report back if I’m able to figure something out. A google search shows scattered instances of it that seem to me point to malloc/free. Way over my head.