Here’s 3 picture of one experiment, the max_memory_allocated rise up quickly in the first or two epoch, and remains almost constant after many epoch. All of sudden, the OOM occurs.

a.png1920×1080 127 KB


a.png1920×1080 192 KB


a.gif786×631 15.6 KB