Hi @Yangmin, I’d like suggest you to have look at this class Instance in Detectron 1 and the Datasampler  1.
The former one also includes dictionaries in a list as input of the reference model.
Basically, I guess you are missing the part of parallelizing your data on multiple devices. You will get some ideas from those two classes.
your_dataloader(dataset, distributed, epoch)  function should look like as follows:

sampler = samplers.distributed.DistributedSampler(dataset=dataset, epoch=epoch) if distributed else None
shuffle = cfg.DATA_LOADER.SHUFFLE if sampler is None else False

loader = torch.utils.data.DataLoader(dataset=dataset,
                                     batch_size=cfg.TRAIN.BATCH_SIZE,
                                     shuffle=shuffle,
                                     num_workers=cfg.DATA_LOADER.NUM_WORKERS,
                                     drop_last=cfg.DATA_LOADER.DROP_LAST,
                                     pin_memory=cfg.DATA_LOADER.PIN_MEMORY,
                                     sampler=sampler,
                                     collate_fn=dataset.sample_collate_train
)
return loader

Hope it helps you to work it out.