thank you @ptrblck I have managed to solve this by setting broadcast_buffers=False It turns out that having it as True is an inplace operation.
but now, I have a massive memory leak issue which I have described here:
  
    
    
    Massive memory leak when using distributed distributed
  
  
    I have a distributed code base I am trying to work with, but, with every epoch I see that my CPU memory increases almost linearly eventually running into OOM on a large 128GB machine :(( 
Without distributed, the code runs fine with no such issues. The isue is exactly describe here: CPU memory gradually leaks when num_workers > 0 in the DataLoader · Issue #13246 · pytorch/pytorch · GitHub 
but the solution posted there, using pyarrow, does not solve my issue - I still have this memory leak. I am…
  


:((