Does the GPU memory build-up remain at those particular points i.e. 8 and 4 GB once they have reached upto those points, or do they increase further?
If you’re not already - you can try using nvidia-smi -l 1 on the terminal to continuously monitor the GPU memory usage