While building PyTorch for your compute capability would speed up the build process, the CUDA context should already only load the device kernels, so I would not assume to see a reduction in memory.
@tlim besides the loaded kernels coming from libraries, you would also need to check the native PyTorch kernels, which are often templated for various dtypes etc. and would also use memory on the device. @eqy profiled the CUDA context size a while ago and could probably add more information.