You can always run nvidia-smi to see if the processes that you launch are the only ones consuming GPU memory.
There may be other system processes that use the GPU, but they usually don’t use more than 100MB normally (On Ubuntu).