Okay, I think I have narrowed down the problem and realise your answer is pertinent @ptrblck. The network takes in images of varying sizes and not a fixed input, causing the memory to fluctuate.
I have managed to train the network on V100 GPU and here are the output from first two(200*2) iterations,

Epoch: [1]  [   0/7925]  eta: 6:09:17  lr: 0.000500  loss: 2.0259 (2.0259)  loss_classifier: 0.6635 (0.6635)  loss_box_reg: 0.0236 (0.0236)  loss_objectness: 0.6907 (0.6907)  loss_rpn_box_reg: 0.6483 (0.6483)  time: 2.7958  data: 1.1561  max mem: 7464
Epoch: [1]  [ 200/7925]  eta: 1:25:06  lr: 0.000500  loss: 1.0027 (1.3102)  loss_classifier: 0.1189 (0.2118)  loss_box_reg: 0.0081 (0.0198)  loss_objectness: 0.5959 (0.6495)  loss_rpn_box_reg: 0.3003 (0.4291)  time: 0.6534  data: 0.0127  max mem: 10348

I have opened an issue 3 this morning, a proposal to remove GeneralisedTransformation as a compulsory transformation to all FasterRCNN models and I believe fixed size inputs would keep the memory from fluctuating.
Thanks again!