Hi
I did not tried to imply that you are using anaconda, but in fact almost all packages in pip repository are generic and you will not find any optimized package there (just generic compiled).
I don’t experience the same problem as you because I build my own software and optimize it to my resources.
I don’t even need to try to setup one box with exact same resources as you have, to replicate your problem,  because I understand you are just using the generic packages in all the pieces of it (cuda, cudnn, pytorch) and complaining about memory inefficiency, instead of looking in how to optimized them.
About your hated question. First try to understand what lapack does, and why magma was created.
Performance on computations and memory efficiency depends on it.
Your main question is not a problem of pytorch. Its a problem of a non optimized setup.