How about
...
if positive_count > 0:
    positive_indexes = (labels != 0).nonzero()[:, 0]
    boxes = boxes[positive_indexes]
    box_target = box_target[positive_indexes]
    box_loss = F.smooth_l1_loss(boxes, box_target)
    box_loss.backward()
    net.step()
...

? You might need to if you are not in a scope and are seeing memory issues, you might need to del things to free up the graph.