I think I might have some clues on what is the problem related to the CUDA Out of Memory error. My dataset has several bounding boxes (>20)  per image and according to this 1 andthis 1, calculation of IOU in the GPU is memory expensive.
In those issues, they suggest reducing batch size. So I though my batch size of 2 was low enough, but after I changed it to 1, the training process did started and has not stopped for 70 epochs. I wanted to implement thesuggestion in the thread 4 on moving IOU calculation to CPU (which can severely slow down training).
Do anyone knows where the IOU calculation code is performed for the detection models? Any comments will be appreciated.