Hi,
My example code is as follows,
the logs in log file includes ‘XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n’.
however,  some GPU memory is still in use and GPU util is 0%.

it seems that process doesn’t exit, or process exist but memory is not cleaned.
is there any solution for this problem?
I used the following code to start 2 processes:
CUDA_VISIBLE_DEVICES=6,7 WORLD_SIZE=2 python -m torch.distributed.launch --nproc_per_node=2 --master_port 44744 train.py 
def runner_full(ctp):
   
	logs = open('log.txt', 'a')
	train(ctp) # in train we will use GPU to train network
	torch.cuda.empty_cache()

    logs.write('XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n')
    logs.flush()
    logs.close()
    return

if __name__ == "__main__":

    ctp = {'a':0, 'b':1, 'c':2}
    runner_full(ctp)