Apologies. I went ahead and check the code. The smallest block size is 1MB. So the memory usage you see is not related to that.
However, it is still very possible that most of 0.5GB is CUDA ctx. The actual size depends on the CUDA version & card type (reference 3). I checked with a 1080Ti & CUDA 8 I have access to, it takes 343MB. And from my memory on a Titan Xp and CUDA 9 it took ~200MB.
Google search shows other people reporting a ctx as large as 500MB: https://stackoverflow.com/questions/12109985/cudasetdevice-allocates-more-than-580-mb-of-global-memory/14101969 3.
You can check by running another CUDA program and see if it also allocates ~500MB on the first CUDA call.
Btw, in your code, do you mean to use memory_cached rather than max_memory_cached?