Thanks @lgusm. This relates to another question I posted a few weeks ago (Browser Crashes with CONTEXT_LOST_WEBGL - #4 by axel 4), I managed to track down the root of that problem and it seems to be that I’m just running out of GPU memory during training. I track and manually dispose every single tensor I use and can confirm they never grow above ~38 (when the batch size is 3), so I’m not sure where the memory consumption is coming from. I also tried setting WEBGL_DELETE_TEXTURE_THRESHOLD to 0 to no avail.
It would be great if @Jason or someone else could shed some light on this.
Thanks!