Hi everyone,
I’m trying to train the  FactorizableNet 7 using multi-GPUs.  My GPUs are Titan Xp.
This network can be trained with the batch size equal to the number of GPUs. As you can see in the below code snip, the main input of the model is a list of images of diffirent sizes, not a tensor. The dataloader and the collate function just return a list of tuples.
for i, sample in enumerate(loader): # (im_data, im_info, gt_objects, gt_relationships)
        # measure the data loading time
        batch_size = len(sample['visual'])
        # measure data loading time
        meters['data_time'].update(time.time() - end, n=batch_size)

        input_visual = [item for item in sample['visual']]
        target_objects = sample['objects']
        target_relations = sample['relations']
        image_info = sample['image_info']
        # RPN targets
        rpn_anchor_targets_obj = [[
                np_to_variable(item[0],is_cuda=False, dtype=torch.LongTensor),
                np_to_variable(item[1],is_cuda=False),
                np_to_variable(item[2],is_cuda=False),
                np_to_variable(item[3],is_cuda=False)
                ] for item in sample['rpn_targets']['object']]

        # compute output
        try:
            raw_losses = model(
                        im_data=input_visual, 
                        im_info=image_info,
                        gt_objects=target_objects,
                        gt_relationships=target_relations,
                       rpn_anchor_targets_obj=rpn_anchor_targets_obj)
.....

The problem is that the time to complete a batch with batch size 1 is 1/8 of time for a batch of size 8. This means the training time is not reduced at all. Memory on the The GPU volatile utilization of all GPUs are very low.
I have asked the author, but it seems that he does not have time to answer questions.
What can I do now to reduce training time?
Thank you.