I have found issue and updated my code. I have posted working solution. Thank you for the support.

@odats great !
However, seeing your DatasetRandom implementation, it seems a bit weird to have the same random data each epoch as you said it in the very begining:

Without setting a random seed the data loader returns the same random data for each epoch:
epoch 1: worker1->[2], worker2->[2], epoch 2:  worker1->[2], worker2->[2],...

Probably, somewhere you should have some unwanted random seed synchonization. Which ignite version you are using, btw ?