You can also try passing broadcast_buffers=False to DDP, although note that this will disable buffer synchronization which might affect model quality if you wanted to ensure all buffers are synchronized.