The nightlies can be used with CUDA11.3. For CUDA11.4 we need to refactor the build pipeline a bit as the cudatoolkit binary was changed and we decided to work on it after the PyTorch 1.10.0 release.
In the meantime you can of course build from source using CUDA 11.4.