Hi ,
I am trying to speed up the training process and use the resources optimally. My GPU utilization is low but my GPU memory is full, I am really not able to understand, where is the bottleneck and how to approach it.
Below are my system results -
image2732×1332 249 KB
image2756×1352 355 KB
image2742×1352 368 KB
I am training the 10k images which are resized from size 2048,2048 to size 512,512.
I want to train it further on larger resolution too but it is maxing out at 1024,1024. But, my GPU utilisation is really low.
Batch size is 20.
Is there any way to allocate less memory and maximise GPU utilisation.
Also, any other suggestions to optimise it is hugely appreciated.