@mrshenli,Sorry,I can not reply you in time.
Now I found the other 1.5GB comes from system.There is 1.5GB consumption before I start this job so I think cache consumption is right.
I use the psutil module to test the process memory footprint.The result just like the following photo:
image1096×265 13.4 KB
(1 node)
image1226×343 16.2 KB
(16 node)
So the process memory footprint per machine does not increase with the total number of machines.However,the total memory footprint per machine increases significantly.
Then I found the process numbers increase significantly with the number of machines.

I set nproc_per_node to 1 so there should be one process in every machine.However,it is not.
There is my command:
bash.sh
MIP=$ip
MPORT=$port
NPROC_PER_NODE=1
HOSTLIST=$hostlist
COMMAND=$HOME/sample.py --epochs=120 --workers=0 --batch-size=1 --print-freq=50 --data=$HOME/datasets/v3
RANK=0
for node in $HOSTLIST; do
        echo $node
        ssh -q $node \
                python -m torch.distributed.launch \
                --nproc_per_node=$NPROC_PER_NODE \
                --nnodes=$SLURM_JOB_NUM_NODES \
                --node_rank=$RANK \
                --master_addr="$MIP" --master_port=$MPORT \
                $COMMAND > "log_v1_"${SLURM_JOB_ID}"_"${RANK}".out" &
        RANK=$((RANK+1))
done

sample.py
....
if __name__ == '__main__':
    if torch.distributed.is_available():
        dist.init_process_group(backend='gloo',init_method='env://',timeout=datetime.timedelta(seconds=600))
        main()

In addition,I do not use CUDA and I train this model in CPUs.