The memory will be occupied after loading the CUDA kernels.
Depending on the build you are using, all libraries might be loaded directly when the context is created (e.g. in the pip wheels as libs are statically linked) or when they are used the first time (e.g. cuDNN via dynamic linking should behave like this).
Regardless of this behavior, if you are using “all libraries” the context size will be the same.