I’ve tried again with today nightly with a totally reproducible command (I suppose you are writing the cache with the same commit of the nightly available wheel):
docker run --rm -it tensorflow/build:latest-python3.9 /bin/bash -c "git clone  --depth 200 https://github.com/tensorflow/tensorflow.git --single-branch  /tf/tensorflow && pip install tf-nightly-cpu && python -c \"import tensorflow as tf; import re; print(re.search('g(\S+)',tf.__git_version__).group(1))\" | GIT_DIR=/tf/tensorflow/.git xargs git reset --hard && cd /tf/tensorflow && bazel --bazelrc=/usertools/cpu.bazelrc build --config=sigbuild_remote_cache  --verbose_failures tensorflow/tools/pip_package:build_pip_package"

But I start to not hit the cache around 5000/6000 actions… e.g.:
 tensorflow/compiler/xla/service/*