I am developing a small utility package to load data to train recurrent nets. The idea is to stream temporally coherent batches. I reuse the IterableDataset with some tweaks to allow to join over multiple streams in the same batch, and to do so with multiple workers:
  
      

      GitHub
  

  
    

etienne87/pytorch-stream-dataloader 1

  Contribute to etienne87/pytorch-stream-dataloader development by creating an account on GitHub.


  

  
    
    
  

  


In order to split the load of the numerous streams accross the workers, i try to distribute the load evenly, but this is not very ideal as some streams might be longer than others.
Ideally, i would like to communicate between different workers to access a single list of streams and be notified of which streams have already been read.
Is there a workaround with some Pytorch Data framework, like a custom sampler, or do i need to hack this myself using multiprocessing library?
EDIT: I did a first version with a multiprocessing.Mutex. It works as expected.