yes, pytorch is needed only to control which process is doing the writing
the rest is cluster dependent.
i/o in local disk is the fastest and the recommended way by admins. writing in network disk or other shared storage is slow and can be extremely slow. in multi-node jobs, they provided a way to dispatch the data across all nodes using srun in a job script but it is not clear how to do it from inside a python code. each process needs to do some i/o operations. it has to be done right in this case. i wrote to admins. no answer yet.

so, yes, this question does not concern pytorch anymore. i will edit the question’s header to clarify.
thanks.