Hi everyone,
I’m implementing the TTNet paper 4. The input tensor of the model consists of N=9 sequential images, so it has a size of (batch_size, 27, H, W).
input_tensor.PNG722×487 79.5 KB
To load an image sequence, I implement the code as below (also here 1)
origin_imgs = []
for img_path_idx, img_path in enumerate(img_path_list):
    origin_imgs.append(load_raw_img(img_path))
origin_imgs = np.dstack(origin_imgs)  # (1080, 1920, 27)

I have trained the model with 4 GPUs using torch.nn.parallel.DistributedDataParallel(), batch_size=32, n_workers = 8.
However, the loading data phase is a bottleneck. This leads to a very slow training process (around 1 hour/epoch).
In the case of N=1, the input tensor has a shape of (batch_size, 3, H, W), training 1 epoch just takes me around 4 minutes.
Please suggest me solutions to boost up the loading data phase.
Thank you.