Hi,
I managed to solve the problem. I still don’t know what went wrong with the torch data loader internally but here is what worked for me.
When I was creating dataset class instance, I was doing something like this
train_dataset = Generator(cfg, mode='Train')

In the init() function of Generator class, i was doing something like this
self.cfg = cfg
self.actual_batchsize = None
self.mode = mode
self.dataset_size = None
self.train_data = []
self.test_data = []
self.val_data = []
self.process_data = ProcessData(cfg)

Where ‘cfg’ was an instance of a class holding some configurations and ‘ProcessData(cfg)’ was also creating an instance of a class.
When I removed ‘cfg’ and ‘ProcessData’ class instances from init_function(), the data loader started working fine even with num_worker>0 and torch multiprocessing.
I dont understand this behavior and I would like to know what is the root cause of this problem?
best,
mvish7