I was able to overcome the issue with CUDA Out of memory error. I did not solve it since I have not yet tried moving IOU calculation to CPU but at least now I can train with batch_size > 1.
In the end I simple reduced the number of anchors created from 5 to 3, and reduced the size of the input images. This allowed me to increase the batch size from 1 to 8.
While doing this, I needed to resize both input images and targets (bounding box, area). Luckily, the detection models released with torchvision 0.3 already include resizing transforms that take care of both images and targets. Please take a look at this thread 20.
I will close this issue now.