Hi,
If this is a one time thing, then this is most likely because it initializes the cuda context on GPU 0.
We discourage the use of this function unless you do have a very good reason to release that memory. Could you explain your usecase for needing to clear the cache and not use CUDA_VISIBLE_DEVICES ?