Not sure what is the reason but I am quite often getting MemoryError exception when using num_workers != 0. Could somebody describe how this process usually works? Does it copy dataset instance (including all its properties) into subprocess? Or does it use threads?
It seems that during the training process the amount of free RAM continues to reduce. I am using a custom dataset that generates images from strokes (Quick Draw Doodles data), and probably the problem is that the dataset doesn’t work well in multitasking setting. Could somebody give an advice on how to implement a multithread ready dataset?