I have just went though the code of Dataloader. The multiprocessing version will prefill 2 * num_workers mini-batches. That is if the whole batch is loaded, the memory consumption should be 2 * 8 * 32  * 500 * 3 * 64 * 64 * 4 / (1024 ** 3) = 12G. This explain the huge memory consumption. To free the additional data, i.e. outside the split, we need to copy the split data to free the pointer to the loaded ndarray. That is, instead of
return {k : v[start:end] for k, v in data.items()}

we should
return {k : v[start:end].copy() for k, v in data.items()}