I am measuring the time by looking at the “iterations per second” measure in the tqdm output and I am measuring GPU memory usage by running nvidia-smi in the terminal. The real “measure of GPU memory use” for me is actually the ability to increase the batch size.
For example, on my machine (running on just 1 rtx 2080 TI) I can run a batch size of 256 in the above script with CHECKPOINT = True but get a CUDA OOM error when I run it with that batch size and CHECKPOINT = False (I even get an OOM with CHECKPOINT = False with batch size of 64, I need to go all the way down to 32 to get it to run). I am totally open to this being the result of some terrible optimization (or lack thereof) in my code that checkpoint is for some reason resolving 
Oh and no, there is nothing else running on this machine, and no other users.