memory_reserved shows the allocated and cached memory. You could use the memory_summary again to check the different allocations.
nvidia-smi shows the overall memory usage. If no other processes are running, please share an executable code snippet to reproduce the memory increase seen via nvidia-smi.