I see. If you tried to actually test memory allocations to system limits and they seem to fail, maybe it’s a CUDA bug. At this point the best thing to try is probably upgrade CUDA versions. I cant think of anything else.
My previous answer was from seeing this behavior on some of my systems and then realizing that the kernel was just caching some pages.