ok. well, i guess it’s like loading a standard c/c++ library. the whole code is loaded into memory, which is a lot in case of pytorch (as you said).
and in case of cuda it might take more space than standard c/c++ as it needs to load the device independent version and possibly a device specific cached version.
still, to me it seems like a big inefficiency (code could be loaded on demand etc). for me (research) that isn’t an issue, but running inference on embedded devices can be impossible due to that.
cheers for the answer 
edit:
I also found this:
https://devtalk.nvidia.com/default/topic/1044191/determine-memory-cuda-context-memory-usage/?offset=5 59
they have a similar problem, although on a different scale (30mb vs 180mb allocation when only instantiating the kernel on different gpus). an nvidia engineer answered that the allocation depends on the gpu (number of SMs). so i guess in my case (rtx 2080) it’s pretty large.