I don’t think that providing the ptx would be enough, as the user would still need to compile the code and thus would need a local CUDA toolkit installation, which doesn’t seem to be a huge benefit compared to building your extension.
The advantage of building wheels would be to avoid the necessity of a local CUDA toolkit and I think you are right that the support matrix is not small in case you want to support different architectures etc.