A disadvantage of using your 3070 for the display is that you would be using GPU memory, which is usually more limited than the host RAM.
E.g. in my plain Ubuntu20.04 setup my GPU is using ~1.2GB for the display output (Xorg, gnome-shell, firefox etc. are reported by nvidia-smi).