But then it won’t let me allocate memory smaller than a page if that page is already partially used, am I correct?

No – memory smaller than a page may use a partially filled page. It just won’t combine multiple partially filled pages together.

Why does it even bother including the small fragments in the free memory counter.

I think it mostly does not count them. For example, in my experience, if you make lots of small allocations, the free memory counter will go decrease after the first small allocation and then remain unchanged for many subsequent allocations as it uses the remaining portion of the page.

How would you recommend I then try to diagnose a situation that lead me to start on this path? My idea was to write a sort of memory mapper which I would bisect/splice my code with and through its printouts find the guilty parts that caused that much fragmentation.

I would just record the sizes of the malloc and free calls to a log. (Edit THCCachingAllocator.cpp).

Does the content of this error come from CUDA or pytorch?

It comes from PyTorch. Cached in this case is cached but not allocated.