BTW @mrshenli the fact that the file descriptor usage is much higher on node 0 is probably due to the fact that the _all_gather function basically constructs a tree with node 0 at the root, hence node 0 always opens N-1 links. I don’t think any action is needed now, but perhaps we should think if there are other approaches that don’t put all that pressure on a single node.