@ptrblck,  regarding your mention of different GPU on CUDA version producing different context memory allocation, I tested a GTX1080 and a GTX2080 (on CUDA 11.1, PyTorch 1.8.1) and it seems that the GTX1080 allocates 771 MiB while the GTX2080 allocates 1177MiB.
I am wondering whether is there some sort of general guides on CUDA, PyTorch, Driver, GPU Versioning to memory needed for context allocation.  This could be a much easier thing to follow than trying out unique compilations.
At the end of the day, I think it seems that writing the processing code in CUDA is the only way to go about for deployment due to VRAM limitation.