title: Docker image missing on Google Cloud Build "No such image", after being pushed and tagged
body: <p>So we have a step which generates a docker image.</p>
  
  <p>The next step uses the image to run a test, but by referring it using a LOCAL tag to avoid communicating with our remote registry (save time and traffic).</p>
  
  <p>cloud build.yaml:</p>
  
  <pre><code>steps:
  
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: bash
    args: ['./build_image.sh','$PROJECT_ID','base-image']
  
  - name: 'base-image:latest'.  #This should ensure there's no need for an "actual" pull
    args: ['-f', './verify_image.ps1']
  </code></pre>
  
  <p>build_image.sh</p>
  
  <pre><code>#!/bin/bash
  
  project=$1
  applicationName=$2
  version=1.0.3
  image="gcr.io/$project/$applicationName:$version"
  local_image="$applicationName:latest"
  
  docker build -t $image .
  
  echo "Tag image to make it locally availiable so we dont have to do a new pull"
  docker tag $image $local_image
  
  docker push $image
  
  # docker run $local_image #works
  </code></pre>
  
  <p>This works most of the time on GCB. But every once in a while step 2 fails with:</p>
  
  <pre><code>Step #2: Already have image (with digest): base-image:latest
  Finished Step #2
  ERROR
  ERROR: build step 2 "base-image:latest" failed: starting container: Error response from daemon: No such image: base-image@sha256:XXXX
  </code></pre>
  
  <p>I cen reproduce this by for ex starting 10 rebuilds of the same commit and then 0-5 of the rebuilds will throw this error.
  We use this construct several places and makes our build process unstable.</p>
  
  <p>GCB uses docker version:  </p>
  
  <blockquote>
    <p>Docker version 19.03.5, build 633a0ea838</p>
  </blockquote>
  
  <p>I have been searching high and low for an explanation to this error but without any luck.</p>
  
  <p>Things I've tried:</p>
  
  <ul>
  <li><p>I have verified that both tags exists and that the digest's are
  correct in step 1 and step 2 for the base image. As you can se step 2
  confirms that the image with digest exists, but in the next line
  where I expect GCB to do a "docker run base-image:latest", for some
  reason, it does not, sometimes.</p></li>
  <li><p>I've tried doing a docker run using the local tag, in step 1 and it works (the image runs), but immediately after step 2 fails with the above message, claiming the image exists but does not.</p></li>
  <li><p>Tested with the official google builder docker image and the latest official docker image. Same result. </p></li>
  </ul>
  
  <p>Every build on GCB, runs in its own vm host, so every step is a docker container running on the same vm. The docker daemon ,I believe is, hosted on the vm and not in the containers. This is why its possible to use the local tag and avoid the pull. But somehow state gets corrupted in the local registry/daemon or the local tag is not registered. Race condition?</p>
  
  <p>In practice we use this setup so that the first step decides which version of the base-image to use in the second step (frameworks, etc) based on the content of the git repository being build, and then in the next step we can simply reference the local image as base-image:latest. This makes our build process more streamlined and enable us to do "dynamic" changes without bothering other devs on purpose.
  We have a lot of products using a very small number of frameworks the similar way, so this makes good sense.</p>
  
  <p>If anyone has any suggestions how to resolve/fix/workaround this issue please help :).</p>
  
  <p>Kind regards
  Christian</p>