Unfortunately, I’m not familiar with Windows and don’t know how to prevent applications from using the GPU memory.
You could use a small GPU just for the video output and use the bigger one for computation, if it’s a workstation and you are not planning to use it e.g. for gaming.