Hi, here is the problem description:
I’m training an IMPALA agent on a GPU with allocated memory 113000MB, and every time I train over 1.1M frames I got this error:
Screen Shot 2020-08-28 at 11.49.58 am1754×580 101 KB
It didn’t show something like “Cuda out of memory error”, but just a RUNTIME Error: Can’t alloc. The feature_extraction model contains a six-layer Conv2d net.
I wonder if my model is too big for scalable training? What can I do to reduce the memory usage?