Here is a code which illustrates my training:
for iter1, data1 in enumerate(dataloader1): # batch size of dataloader1 is small because no DataParallel
   optimizer.zero_grad()
   class_loss1, reg_loss1 = model(input1, output1) # model is retinanet
   total1 = class_loss1 + reg_loss1
   total1.backward()
   writer.add_scalars('Loss', {'Loss1': total1.item()}, iter1)
   del total1
   ...
   acc_loss = 0
   for iter2, data2 in enumerate(dataloader2): size dl2 >> dl1 -> batch size = 1 and we accumulate the gradient manually
       with torch.no_grad():
          scores, classification, predicted = model(input2) # inference
       class_loss2, reg_loss2 = model(input2, output2) # model is retinanet
       total2 = class_loss2 + reg_loss2
       total2.backward()
       acc_loss += total2.detach()
       del total2

   writer.add_scalars('Loss', {'Loss2': acc_loss}, iter1)
   optimizer.step()

Such that there are 2 training procedure per iteration. Given that dataloader2 can not fit into one single batch, I manually backward the gradients and at the end of the iterations over dataloader2 I perform the step.
If I remove the iteration procedure over dataloader2, everything works fine.
@ptrblck can you check this please ?