Based on the code in torch.cuda.__init__ that was actually throwing the error the following check seems to work:
import torch
from torch.cuda import device_count, get_device_capability


def is_cuda_compatible():
    if torch.version.cuda is not None:
        compatible_device_count = 0
        for d in range(device_count()):
            capability = get_device_capability(d)
            major = capability[0]
            minor = capability[1]
            current_arch = major * 10 + minor
            min_arch = min((int(arch.split("_")[1]) for arch in torch.cuda.get_arch_list()), default=35)
            if (not current_arch < min_arch
                    or not torch._C._cuda_getCompiledVersion() <= 9000
                    or not (major >= 7 and minor >= 5)):
                compatible_device_count += 1

        return True
    return False

Not sure if it’s 100% correct but putting it out here for feedback and in case anybody else needs it. Will be PRing into torch itself later since it seems like the kind of functionality it ought to have.