Actually, I tried to run the training entirely on CPU. When I started the training on CPU, I got the following error.
Traceback (most recent call last):
  File "multi_gpu_FRCNN.py", line 150, in <module>
    output = model(x_tr,y_tr)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 71, in forward
    detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 763, in forward
    class_logits, box_regression, labels, regression_targets)
  File "/opt/conda/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 36, in fastrcnn_loss
    classification_loss = F.cross_entropy(class_logits, labels)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 4 is out of bounds.

This is when I rectified the num_classes to 5 instead of 4  with  background as a class which solved my problem.
I was using the function below to get boxes and labels
def get_boxes_and_labels(label_file,class_map):

    target = {}
    target["boxes"] = []
    target["labels"] = []
    for anno in label_file:

        if "tag" in anno.keys() and anno["tag"] in class_map:
            coord = []
            coord.append(anno["position"]["x"])
            coord.append(anno["position"]["y"])
            coord.append(coord[0] + anno["size"]["x"])
            coord.append(coord[1] + anno["size"]["y"])

            target["boxes"].append(coord)
            target["labels"].append(class_map[anno["tag"]])

    target["boxes"] = np.array(target["boxes"],dtype="float32")
    target["labels"] = np.array(target["labels"])
    return target


The class map was   class_map = {"freezer":1,"Chiller":2,"SLAB":3,"Vegbox":4} .