Hi @ptrblck, yes, you got it correct. Another detail I forgot to mention is that this only happens in multi GPU training. Single GPU runs never fail.
Regarding shared memory, thank you for the suggestion. I am on it.