Hi,
These 500MB are most likely just the memory used by the CUDA initialization. So there is not way to remove it unless you kill the process.
It seems that the model is only stored in your first process 34296 and the others are using it as expected but just the cuda initialization state is taking a lot of memory