So I have timed the printing of paths from the script above. On a tiny dataset of 51 images on my macbook, the times are output numbers on the right of the file paths:
('mults/train/0/10001.ndpi/40x/40x-236247-10154-18944-5376.png',) 0.0
('mults/train/0/10001.ndpi/40x/40x-236247-10152-18432-5376.png',) 0.0
('mults/train/0/10001.ndpi/40x/40x-236247-10155-19200-5376.png',) 9.5367431640625e-07
('mults/train/0/10001.ndpi/40x/40x-236247-10151-18176-5376.png',) 1.1920928955078125e-06
('mults/train/0/10001.ndpi/40x/40x-236247-10153-18688-5376.png',) 1.1920928955078125e-06
('mults/train/0/1234.ndpi/40x/40x-236247-16658-86528-8704.png',) 9.5367431640625e-07
('mults/train/0/1234.ndpi/40x/40x-236247-16656-86016-8704.png',) 9.5367431640625e-07
('mults/train/0/1234.ndpi/40x/40x-236247-16655-85760-8704.png',) 9.5367431640625e-07
('mults/train/0/1234.ndpi/40x/40x-236247-16657-86272-8704.png',) 1.1920928955078125e-06
('mults/train/0/1234.ndpi/40x/40x-236247-16654-85504-8704.png',) 1.9073486328125e-06
('mults/train/1/5678.ndpi/40x/40x-236247-16635-80640-8704.png',) 9.5367431640625e-07
('mults/train/1/5678.ndpi/40x/40x-236247-16637-81152-8704.png',) 9.5367431640625e-07
('mults/train/1/5678.ndpi/40x/40x-236247-16638-81408-8704.png',) 9.5367431640625e-07
('mults/train/1/5678.ndpi/40x/40x-236247-16634-80384-8704.png',) 0.0
('mults/train/1/5678.ndpi/40x/40x-236247-16636-80896-8704.png',) 0.0
('mults/train/1/10001.ndpi/40x/40x-236247-10142-15872-5376.png',) 9.5367431640625e-07
('mults/train/1/10001.ndpi/40x/40x-236247-10150-17920-5376.png',) 9.5367431640625e-07
('mults/train/1/10001.ndpi/40x/40x-236247-10154-18944-5376.png',) 0.0
('mults/train/1/10001.ndpi/40x/40x-236247-10152-18432-5376.png',) 9.5367431640625e-07
('mults/train/1/10001.ndpi/40x/40x-236247-10155-19200-5376.png',) 9.5367431640625e-07
('mults/train/1/10001.ndpi/40x/40x-236247-10151-18176-5376.png',) 9.5367431640625e-07
('mults/train/1/10001.ndpi/40x/40x-236247-10153-18688-5376.png',) 1.1920928955078125e-06
('mults/train/1/1234.ndpi/40x/40x-236247-16658-86528-8704.png',) 9.5367431640625e-07
('mults/train/1/1234.ndpi/40x/40x-236247-16656-86016-8704.png',) 0.0


The much larger dataset of 150k samples on a separate linux server:
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-170637-97024-63232.png',) 7.152557373046875e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-133769-50944-51200.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-82090-80896-33536.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-74575-19712-30976.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-226275-81408-81664.png',) 2.384185791015625e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-172632-51712-64000.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-82388-23552-33792.png',) 7.152557373046875e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-150970-73984-56832.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-188390-69632-69120.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-216258-61440-78336.png',) 7.152557373046875e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-226105-57088-81664.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-17811-39424-9216.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-221670-72448-80128.png',) 9.5367431640625e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-77732-62464-32000.png',) 7.152557373046875e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-258005-84480-92672.png',) 7.152557373046875e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-223683-36864-80896.png',) 7.152557373046875e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-176327-25600-65280.png',) 1.1920928955078125e-06
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-100073-33536-39936.png',) 9.5367431640625e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-163497-84992-60928.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-127859-80896-49152.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-167504-97792-62208.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-263262-45568-94976.png',) 4.76837158203125e-07
('/vol/vssp/cvpnobackup/scratch_4weeks/taran/sample/train/1/F17-013461/40x/40x-F17-013461-205135-9984-74752.png',) 4.76837158203125e-07


They appear to be similar. I’m wondering if the speed of retrieving the batches is the issue 
Again, no multiprocessing error anymore…which is good…