We are not backporting new library support for old releases. The already released PyTorch versions are supporting the CUDA toolkits which were supported at that time.
With that being said, if no changes e.g. between CUDA 11.3 and 11.4 were needed, you might be able to use the newer CUDA toolkit, but there is no guarantee.