I’m not familiar enough with the Python memory management and garbage collection mechanism to be able to explain the effect you are seeing properly. Since rss returns the “non-swapped physical memory” of the process, you would not only see the required memory used by the tensor allocation but also the memory usage by the process itself, loaded libraries, as well as all other objects.
My guess would be that the garbage collection kicks in at different intervals and might free some memory, so you could check if playing around with its thresholds might change the behavior.