Answers: <p>GitLab has some docs on how to speed up builds, and one of the suggestions does involve caching runner images.</p>
  <p><a href="https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers" rel="nofollow noreferrer">https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers</a></p>
  <h2>Use a proxy for containers</h2>
  <p>You can speed up the time it takes to download Docker images by using:</p>
  <ul>
  <li>The GitLab Dependency Proxy or</li>
  <li>A mirror of the DockerHub Registry</li>
  <li>Other open source solutions</li>
  </ul>
  <p>There are other non-container related suggestions too, as well as information on how to use these up at that link.</p>
  <p>You can also host your own runners, in which case you can run them on docker or just centos, etc.  I assume, in that case, that you could already have the images available on the runner node to avoid new pulls.  We run ours in kubernetes and are looking at harbor as an option to speed these things up too, but we haven't tested it yet.  I imagine self-hosted k8s and harbor would be similar to GitLab using its own dependency proxy solution.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-02-26T22:29:20.380Z
Answers: My Runner(s) are actually self hosted. They run on a Server as Docker-Runner. If I understand correctly instead of linking a proxy somewhere I could save the images (In my example python:3.9) locally inside the gitlab runner node and reference them?
Answers: MaKaNu
Answers: CC BY-SA 4.0
Answers: 2022-02-27T02:28:41.723Z
Answers: If you're using a single-server docker setup, I wouldn't expect it to re-pull images it already had.  The underlying docker daemon should recognize it already has an image/relevant layers and should just reuse it without going to docker-hub/etc.  That's how docker works.
Answers: John Humphreys
Answers: CC BY-SA 4.0
Answers: 2022-02-27T15:00:35.747Z
Answers: So it seems I am doing something wrong in the pipeline, because I just use one Dockerized-Runner on a single server, but both jobs are pulling the python:3.9 image.
Answers: MaKaNu
Answers: CC BY-SA 4.0
Answers: 2023-06-05T10:50:52.970Z
Answers: The docs you link to involve caching in a registry (even if local) as opposed to the local docker cache, which is what I'm looking for, and what I assume the question is asking for as well.
Answers: Chris Stryczynski
Answers: CC BY-SA 4.0
Answers: 2022-02-26T19:09:27.697Z
Answers: 2022-02-26T19:09:27.697Z
Answers: John Humphreys
Body: <p>I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:</p>
  <pre class="lang-bash prettyprint-override"><code>Running with gitlab-runner 14.8.0 (565b6c0b)
    on Shared gitlab-runner on ARWS. 3yyScM1M
  Preparing the &quot;docker&quot; executor 00:03
  Using Docker executor with image python:3.9 ...
  Pulling docker image python:3.9 ...
  Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
  Preparing environment
  .
  .
  .
  </code></pre>
  <p>This part seems to take most of the pipeline time. At the moment not much is happening in both jobs.
  I already started to implement caching in the pipelinbe:</p>
  <pre class="lang-yaml prettyprint-override"><code>image: python:3.9
  
  stages:
    - test
    - build
  
  variables:
    PIP_CACHE_DIR: &quot;$CI_PROJECT_DIR/.cache/pip&quot;
  
  cache:
    key:
      files:
        - poetry.lock
        - .gitlab-ci.yml
      prefix: ${CI_JOB_NAME}
    paths:
      - .venv
      - .cache/pip
  
  before_script:
    - python --version  # For debugging
    - pip install poetry
    - poetry config virtualenvs.in-project true
    - poetry install
    - source `poetry env info --path`/bin/activate
  
  testing:
    tags: 
      - docker
      - linux
    stage: test
    script:
      - echo &quot;Launching Job 'testing' in stage 'test'&quot;
      - pytest -v --junitxml=report.xml
    artifacts:
      when: always
      reports:
        junit: /builds/$CI_PROJECT_PATH/report.xml
  
  building:
    tags:
      - docker
      - linux
    stage: build
    script:
      - echo &quot;Launching Job 'building' in stage 'build'&quot;
  </code></pre>
  <p>And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?</p>
  
Tags: <docker><gitlab><gitlab-ci-runner><cache>
Title: How to caching the docker images?
UserDisplayName: MaKaNu
ThreadText: # How to caching the docker images?
  
  (Asked by: username_0 on 2022-02-26)
  
  I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:
  
  ```bash
  Running with gitlab-runner 14.8.0 (565b6c0b)
    on Shared gitlab-runner on ARWS. 3yyScM1M
  Preparing the "docker" executor 00:03
  Using Docker executor with image python:3.9 ...
  Pulling docker image python:3.9 ...
  Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
  Preparing environment
  .
  .
  .
  
  ```
  
  This part seems to take most of the pipeline time. At the moment not much is happening in both jobs.
  I already started to implement caching in the pipelinbe:
  
  ```yaml
  image: python:3.9
  
  stages:
    - test
    - build
  
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  
  cache:
    key:
      files:
        - poetry.lock
        - .gitlab-ci.yml
      prefix: ${CI_JOB_NAME}
    paths:
      - .venv
      - .cache/pip
  
  before_script:
    - python --version  # For debugging
    - pip install poetry
    - poetry config virtualenvs.in-project true
    - poetry install
    - source `poetry env info --path`/bin/activate
  
  testing:
    tags: 
      - docker
      - linux
    stage: test
    script:
      - echo "Launching Job 'testing' in stage 'test'"
      - pytest -v --junitxml=report.xml
    artifacts:
      when: always
      reports:
        junit: /builds/$CI_PROJECT_PATH/report.xml
  
  building:
    tags:
      - docker
      - linux
    stage: build
    script:
      - echo "Launching Job 'building' in stage 'build'"
  
  ```
  
  And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?
  
  **Tags:** docker, gitlab, gitlab-ci-runner, cache
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-26. Score: 0
  
  GitLab has some docs on how to speed up builds, and one of the suggestions does involve caching runner images.
  
  <https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers>
  
  ## Use a proxy for containers
  
  You can speed up the time it takes to download Docker images by using:
  
  * The GitLab Dependency Proxy or
  * A mirror of the DockerHub Registry
  * Other open source solutions
  
  There are other non-container related suggestions too, as well as information on how to use these up at that link.
  
  You can also host your own runners, in which case you can run them on docker or just centos, etc. I assume, in that case, that you could already have the images available on the runner node to avoid new pulls. We run ours in kubernetes and are looking at harbor as an option to speed these things up too, but we haven't tested it yet. I imagine self-hosted k8s and harbor would be similar to GitLab using its own dependency proxy solution.
  
  > **Comments:**
  > **username_0**: My Runner(s) are actually self hosted. They run on a Server as Docker-Runner. If I understand correctly instead of linking a proxy somewhere I could save the images (In my example python:3.9) locally inside the gitlab runner node and reference them?  
  > **username_1**: If you're using a single-server docker setup, I wouldn't expect it to re-pull images it already had.  The underlying docker daemon should recognize it already has an image/relevant layers and should just reuse it without going to docker-hub/etc.  That's how docker works.  
  > **username_0**: So it seems I am doing something wrong in the pipeline, because I just use one Dockerized-Runner on a single server, but both jobs are pulling the python:3.9 image.  
  > **username_2**: The docs you link to involve caching in a registry (even if local) as opposed to the local docker cache, which is what I'm looking for, and what I assume the question is asking for as well.  
  
  ---
  
  