I am training my models in a local machine with a GPU available. My particular application is in RL, where I have multiple workers working in parallel (implemented with python3 multiprocessing) collecting experience using my torch models on the cpu since only forward passes are needed. Periodically I also have a main process, that trains an instance of the model on the GPU and then updates the weights of the worker models.
The problem that I  face is that, also the worker processes take around 800 MBs of memory on the GPU (I assume for the CUDA kernels) even though they never access the GPU.

Screenshot from 2021-11-04 12-39-45722×567 69.2 KB

The screenshot shown, presents an exemple view of nvidia-smi. In this example I am using a simple MLP model (2MB of memory needed), and I am using 10 worker processes, which can be seen to take 797 MBs of memory each, and the master process (id 1707716) that takes 799 ( 2 MB more because of the model memory).
I am not adding portions of the code, as It is pretty complex and spread around multiple files but I wanted to ask if there is a way to avoid loading the kernel in the GPU memory for every process using torch or if there is a way to free it.