Thanks for the detailed information. This is weird. This is the first time that we saw the memory footprint per machine increases significantly with the total number of machines.

I guess the memory is used to load the disk data.I tried to use a small datasets
which only contains 40 samples(Every sample is 16MB).And in this case,the cache consumption is about 2.5GB.

Q1: This does not add up. The model is 10MB, and the dataset is 40X16MB = 640MB.  I assume you do not use CUDA, as you mainly concerned about CPU memory. In this case, I would assume the total memory consumption to be less than 1GB. Do you know where does the other 1.5GB come from?
Q2: Can you try to the following to see if it is indeed used by the process instead of some os cache.
import psutil

process = psutil.Process(os.getpid())

for _ in get_batch():
    ....
    # print this in every iteration
    print("Memory used: ", process.memory_info().rss)

Q3: Do you have a minimum reprodueable example that we can investigate locally?
Q4: BTW, how do you launch your distributed training job? Are you using torch.distributed.launch. If so, could you please share the command you are using?