I did step through the code in debugger and it looks like its able to follow the similar structure to the original code loading the model and data while training. I’m able to load the models trained in original implementation into my pipeline and run some eval operations on it. Also able to load the pretrained weights provided for mobilenet1 and mobilelet2 were loading for training. But I can try doing it once again by isolating and checking them as you have suggested.


The main vision folder which contains various modules for model, data and losses is almost same in my integration. The only changes that I made were :




Changed the data loading part to load images and labels from my custom dataset file. The prepossessing applied on the images and labels loaded from my dataset is the same as the original implementation.


Moved some of the tensors to ‘cpu’ device  as it was causing problem in dataloader it tried to do multiprocessing in cuda device when num_workers>0. After moving the tesnor to cpu it prevented the error as cpu processes were spawned for dataloading and preprocessing. Below is the diff from this file 1


@@ -141,7 +143,7 @@
 class MatchPrior(object):
     def __init__(self, center_form_priors, center_variance, size_variance, iou_threshold):
         self.center_form_priors = center_form_priors
-        self.corner_form_priors = box_utils.center_form_to_corner_form(center_form_priors)
+        self.corner_form_priors = box_utils.center_form_to_corner_form(center_form_priors).to('cpu')
         self.center_variance = center_variance
         self.size_variance = size_variance
         self.iou_threshold = iou_threshold
@@ -149,11 +151,13 @@
     def __call__(self, gt_boxes, gt_labels):
         if type(gt_boxes) is np.ndarray:
             gt_boxes = torch.from_numpy(gt_boxes)
+            gt_boxes = gt_boxes.to('cpu')
         if type(gt_labels) is np.ndarray:
             gt_labels = torch.from_numpy(gt_labels)
+            gt_labels = gt_labels.to('cpu')
         boxes, labels = box_utils.assign_priors(gt_boxes, gt_labels,
                                                 self.corner_form_priors, self.iou_threshold)
-        boxes = box_utils.corner_form_to_center_form(boxes)
+        boxes = box_utils.corner_form_to_center_form(boxes).to('cpu')
         locations = box_utils.convert_boxes_to_locations(boxes, self.center_form_priors, self.center_variance, self.size_variance)
         return locations, labels


Another change for this was made to this file: /vision/ssd/config/mobilenetv1_ssd_config.py

@@ -20,4 +20,4 @@
 ]
 
 
-priors = generate_ssd_priors(specs, image_size)
\ No newline at end of file
+priors = generate_ssd_priors(specs, image_size).to('cpu')
\ No newline at end of file


Yes, the softmax function was used in debugging. The softmax is applied only when the model is run in test mode to get the probability outputs. I applied softmax on the individual feature maps which were output from confidence headers of various layers, which is when I discovered this discrepancy between the outputs of the model trained in the original implementation and the one trained in my pipeline.