Monitoring the memory looks as if pytorch already allocated 23GB before throwing the OOM error. Let along that the machine I am running this on has 32GB of RAM available. (If you see the graph in my previous post it shows the memory allocated on the machine at the time of the OOM error)
Unfortunately I’d love to run the CNN on my GPU but the memory is a more important factor there. I have two GTX 1070 with 8GB ram each but I haven’t figured if I can use the memory of both of them in the same training