I’m training a segmentation DRN network, and I’ve encountered a weird gpu memory problem, which I have no idea how to approach.
If I use batch sizes in a specific range, in my case 16-28, I get a CUDA out of memory crash. Whereas when using larger batches of 32,64 or 128 everything works fine. Same for using batch sizes < 16.
This just makes no sense to me, and is especially annoying since for a more complex training procedure with two networks the maximum gpu memory capacity falls within the problematic batch-size range, and I’m forced to use very small batches.
Any ideas how to fix this?
I’m using torch version 0.4.1 with cuda 9.0.176, cudnn version 7102
on a Tesla K80 gpu with nvidia driver 390.12.