@colesbury, your detailed reply is extremely helpful and insightful, thank you!
So as you explained I need to track fragmentation at a much lower  level. But then it won’t let me allocate memory smaller than a page if that page is already partially used, am I correct?
Why does it even bother including the small fragments in the free memory counter. In addition to the current free memory it really should have a total real free mem counter, which would exclude accounting for any partially filled pages.
How would you recommend I then try to diagnose a situation that lead me to start on this path?
RuntimeError: CUDA out of memory. 
Tried to allocate 350.00 MiB 
(GPU 0; 7.93 GiB total capacity; 5.73 GiB already allocated; 
324.56 MiB free; 1.34 GiB cached)

My idea was to write a sort of memory mapper which I would bisect/splice my code with and through its printouts find the guilty parts that caused that much fragmentation.
Does the content of this error come from CUDA or pytorch? It doesn’t look like pytorch, since its memory counters are always: allocated <= cached. So then it’s unclear what that ‘cached’ section mean.
Based on your explanation that means that there is a helluva lot of small fragments sprinkled through the RAM, so that it can’t allocate that little memory (0.35GB) in so much available (cached+free) 1.7GB of RAM.