Hi,
I have read the related posts about best practices for usage on GPU, and this question is different.
I am tuning my batch size such that the largest batch uses around 90% of GPU memory. Thus, some batches will use close to 90% while other (most use around  50%)
I am noticing that if 2 large batches (both using close to 90%)  are processed (i.e. one forward prop and batch prop) consecutively it crashes with out of memory error. I suspect because the GPU is unable to clear the memory of the previous batch. However, if large batches and interspersed with smaller batches it seems to work fine.
Is there a rule of thumb to GPU utilization for such purposes, i.e. an optimal memory usage that works with the garbage collection?