Solved by juanmed in post #6 
              
              
                I was able to overcome the issue with CUDA Out of memory error. I did not solve it since I have not yet tried moving IOU calculation to CPU but at least now I can train with batch_size > 1. 
In the end I simple reduced the number of anchors created from 5 to 3, and reduced the size of the input imag…