The PyTorch conda binaries and pip wheels ship with their own CUDA runtime, so you could install different versions using virtual environments on the same system. Your local CUDA toolkit would be used, if you build PyTorch from source or compile a custom CUDA extension.
Since you suspect that framework changes are responsible for the increased memory usage, this would allow you to compare different release versions.