Finally, I finished the training procedure successfully!
I learn the chunk part from @ptrblck, hdf5 part from @rasbt, and the sampler part from @Egor_Kraev, thanks for all of you guys!
There are two problems I have to mention, in case of any other people might cost time on them.

KeyError of unable to open hdf5 file


[… in h5py.h5o.open\nKeyError: ‘Unable to open object (bad object header version number)’\n’]

It’s easy to get this error when the num_workers is greater than 2, but even num_workers=1 can get this error except the frequency is much lower. It might be an h5py or hdf5 problem, rerun the code can bypass it.

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes’ failed


[return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes’ failed.  at /opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/THNN/generic/ClassNLLCriterion.c:87]

If you run on gpu, it might be a “triggle device side … error”.

/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THCUNN/ClassNLLCriterion.cu:57: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [28,0,0] Assertion t >= 0 && t < n_classes failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1503970438496/work/torch/lib/THC/generic/THCStorage.c line=32 error=59 : device-side assert triggered

This is a very weird error since I’m definitely sure my target is in [0, n_classes-1], actually just 0 and 1, and also, this is not an always error. Sometimes everything is fine but sometimes it will show up, just like error 1. The only place I suspect is that the target is int32 in hdf5 on the contrary int64 in LongTensor. So I print them out and was shocked by the contents. Some batches can print out 0 and 1 but the error batch print out 0.0…0e0 and 1.12…6e9 something like that, before I cast them. I’m stuck on this error for a long time until I remove all num_workers setting. Besides, I found the smaller the batch size, the lower frequency this error show up.
I think the hdf5/h5py might have some multi-processing reading bugs which is far beyond my ability. So I have to sacrifice the training time to get the output model.