Hi,
My program implements sharing GPU models/tensors in a class by multiple processes. Below is an MWP:
import time
import torch
import multiprocessing as mp
import torch.multiprocessing as tmp


def work(rank, t):
    print("Working")
    time.sleep(10)
    return


class Tester(object):
    def __init__(self, num_proc):
        self.device = 'cuda:1'
        self.tensor = torch.zeros(1).to(self.device)
        self.np = num_proc

    def run(self):
        procs = [mp.Process(target=work, args=(rank, self,)) for rank in range(self.np)]
        [p.start() for p in procs]
        [p.join() for p in procs]
        # tmp.spawn(work, nprocs=self.np, args=(self,))
        

def main():
    t = Tester(4)
    t.run()


if __name__ == '__main__':
    mp.set_start_method('spawn', force=True)
    main()

Essentially, the processes are sharing a tensor which is on ‘cuda:1’. On starting the program, gpustat looks like the following:
Screenshot from 2020-02-13 16-08-402086×370 72.4 KB

However, just before the exit, gpustat shows the following memory use:

Screenshot from 2020-02-13 16-08-462122×451 122 KB

Now, in my actual implementation, wherein I have multiple processes sharing models on each of the GPUs in the system, this results in not enough memory on ‘cuda:0’ and some cudNN errors.
I checked if torch.multiprocessing wrapper, instead of python multiprocessing, has the same issue; and, it does.
What exactly is going wrong here?
Thank you.