My dataloader is extremely slow on first epoch. I have tried to go through all the solutions and threads posted online and almost nothing helps. These are the things I did:

Tested on SSD which also has an NVMe support. The data transfer rate is extremely fast.
Turned off pin_memory, doesn’t help.
Moved from reading one HDF5 file to individual hdf5 files.
Removed zipping hdf5 data.

My model is very small for practical purposes and the batch size is also small. But as the amount of data increases the dataloader gets slower and slower on the first epoch. Is there any solution to this?
Dataloader link:

  
      github.com
  
  
    CGL-Deeplearning/FRIDAY/blob/dev_v0.1_w10_resnet/modules/core/dataloader.py 17
import os
import numpy as np
import pandas as pd
from torch.utils.data import Dataset
import h5py
import torch
import pickle


class SequenceDataset(Dataset):
    """
    Arguments:
        A CSV file path
    """

    def __init__(self, csv_path, transform=None):
        data_frame = pd.read_csv(csv_path, header=None, dtype=str)
        # assert data_frame[0].apply(lambda x: os.path.isfile(x.split(' ')[0])).all(), \
        #     "Some images referenced in the CSV file were not found"
        self.transform = transform


  This file has been truncated. show original

  
  
    
    
  
  


Train script:

  
      github.com
  
  
    CGL-Deeplearning/FRIDAY/blob/dev_v0.1_w10_resnet/train.py 12
import argparse
import os
import sys
import time
import random

import torch
import torch.nn.parallel
import torchnet.meter as meter
import torch.nn as nn
from tqdm import tqdm
from torch.utils.data import DataLoader
from torchvision import transforms
from modules.core.dataloader import SequenceDataset
from modules.models.ModelHandler import ModelHandler
from modules.handlers.TextColor import TextColor
"""
Train a model and save the model that performs best.

Input:


  This file has been truncated. show original