Answers: <p>I highly recommend using <a href="https://docs.npmjs.com/cli/ci.html" rel="nofollow noreferrer">npm ci</a> instead of <code>npm install</code>. The slow part of installing npm modules is figuring out everything that needs to be installed. If you commit your package-lock.json and use <code>npm ci</code>, it skips that part entirely and just looks at your lockfile to download what's needed. It's significantly faster.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-09-24T15:39:58.007Z
Answers: What are your thoughts on npm install with `--prefer-offline` ? I was using npm ci but switched to npm i with the usage of prefer offline. As of yet I don't have an opinion on either - other that npm ci FEELS better due do the use of the keyword `ci`
Answers: Jacksonkr
Answers: CC BY-SA 4.0
Answers: 2020-09-24T18:00:07.727Z
Answers: I've never used `--prefer-offline`. In fact I didn't know it existed before your comment. I think the main benefit of `npm ci` is the fact that you know exactly what will be installed and there are no changes made to package.json. It also deletes node_modules before installing packages so you know it's a clean slate every time.
  
  I've not yet seen issues with `npm ci` slowness, but you could use `--prefer-offline` with `npm ci`. Here's an example from the gitlab docs:  https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies
Answers: bradym
Answers: CC BY-SA 4.0
Answers: 2020-09-25T12:51:43.917Z
Answers: Funny you should mention that gitlab doc - those docs are where I pulled most my current implementation from. It seems like the issue now is that compiling / extracting the cache takes longer than fetching all the npm packages.
Answers: Jacksonkr
Answers: CC BY-SA 4.0
Answers: 2020-09-25T18:51:46.917Z
Answers: Circling back - ultimately I deemed my `prepare` job to be doing double work. I took `prepare` out which cut my workload in half. I don't know I'd call that optimized but it's certainly much better.
Answers: Jacksonkr
Answers: CC BY-SA 4.0
Answers: 2020-09-26T00:23:59.570Z
Answers: Ahh, yes. I don't think I noticed you were doing both. And optimization is relative, so yes, it's optimized. :D
Answers: bradym
Answers: CC BY-SA 4.0
Answers: 2020-09-24T00:33:51.750Z
Answers: 2020-09-24T00:33:51.750Z
Answers: bradym
Body: <p>I have a gitlab yaml I've been working on for a bit and while everything works it seems like each job takes quite a bit longer than it should. I tend to think the culprit is that npm files are fetched every time rather than sharing the <code>node_modules</code> cache between jobs.</p>
  <p>Short of creating my own docker image, how can I optimize this yaml in a way that reduces the bottlenecks ?</p>
  <p><strong>gitlab .yaml</strong></p>
  <pre><code>image: node:10.19.0 # https://hub.docker.com/_/node/
  # image: node:latest
  
  cache:
    # untracked: true
    key: my-project
    # key: ${CI_COMMIT_REF_SLUG} # per branch
    # key:
    #   files:
    #     - package-lock.json # only update cache when this file changes (not working) @jkr
    paths:
      - .npm/
      - node_modules
      - build
  
  stages:
    - prepare # prepares builds, makes build needed for testing
    - test # uses test:build specifically @jkr
    - build
    - deploy
  
  # before_install:
  
  before_script:
    - npm ci --cache .npm --prefer-offline
  
  prepare:
    stage: prepare
    needs: []
    script:
      - npm install
  
  test:
    stage: test
    needs: [prepare]
    except:
      - schedules
    tags:
      - linux
    script:
      - npm run build:dev
      - npm run test:cicd-deps
      - npm run test:cicd # runs puppeteer tests @jkr
    artifacts:
      reports:
        junit: junit.xml
      paths:
        - coverage/
  
  build-staging:
    stage: build
    needs: [prepare]
    only:
      - schedules
    before_script:
      - apt-get update &amp;&amp; apt-get install -y zip
    script:
      - npm run build:staging
      - zip -r build-staging.zip build
    # cache:
    #   paths:
    #     - build
    #   &lt;&lt;: *global_cache
    #   policy: push
    artifacts:
      paths:
        - build-staging.zip
  
  deploy-serv1:
    stage: deploy
    needs: [build-staging]
    tags: [linux]
    only:
      - schedules
    #   # - branches@gitlab-org/gitlab
    before_script:
      - apt-get update &amp;&amp; apt-get install -y lftp
    script:
      # temporarily using 'verify-certificate no'
      # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
      # variables do not work with 'single quotes' unless they are &quot;'surrounded by doubles'&quot;
      - lftp -e &quot;set ssl:verify-certificate no; open myserver.org; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye&quot;
    # environment:
    #   name: staging
    #   url: http://dev.myserver.org/projects/my-project/build
    when: manual
    allow_failure: true
  
  build-production:
    stage: build
    needs: [prepare]
    only:
      - schedules
    before_script:
      - apt-get update &amp;&amp; apt-get install -y zip
    script:
      - npm run build:staging
      - zip -r build-prod.zip build
    # cache:
    #   paths:
    #     - build
    #   &lt;&lt;: *global_cache
    #   policy: push
    artifacts:
      paths:
        - build-prod.zip
  
  deploy-serv2:
    stage: deploy
    needs: [build-production]
    tags: [linux]
    only:
      - schedules
      # - master
    before_script:
      - apt-get update &amp;&amp; apt-get install -y lftp
    script:
      - sh deploy-prod
    environment:
      name: production
      url: http://www.my-project.org
    when: manual
    allow_failure: true
  </code></pre>
  
Tags: <continuous-integration><yaml>
Title: devops - Optomizing gitlab cicd pipeline's caching
UserDisplayName: Jacksonkr
ThreadText: # devops - Optomizing gitlab cicd pipeline's caching
  
  (Asked by: username_0 on 2020-09-23)
  
  I have a gitlab yaml I've been working on for a bit and while everything works it seems like each job takes quite a bit longer than it should. I tend to think the culprit is that npm files are fetched every time rather than sharing the `node_modules` cache between jobs.
  
  Short of creating my own docker image, how can I optimize this yaml in a way that reduces the bottlenecks ?
  
  **gitlab .yaml**
  
  ```
  image: node:10.19.0 # https://hub.docker.com/_/node/
  # image: node:latest
  
  cache:
    # untracked: true
    key: my-project
    # key: ${CI_COMMIT_REF_SLUG} # per branch
    # key:
    #   files:
    #     - package-lock.json # only update cache when this file changes (not working) @jkr
    paths:
      - .npm/
      - node_modules
      - build
  
  stages:
    - prepare # prepares builds, makes build needed for testing
    - test # uses test:build specifically @jkr
    - build
    - deploy
  
  # before_install:
  
  before_script:
    - npm ci --cache .npm --prefer-offline
  
  prepare:
    stage: prepare
    needs: []
    script:
      - npm install
  
  test:
    stage: test
    needs: [prepare]
    except:
      - schedules
    tags:
      - linux
    script:
      - npm run build:dev
      - npm run test:cicd-deps
      - npm run test:cicd # runs puppeteer tests @jkr
    artifacts:
      reports:
        junit: junit.xml
      paths:
        - coverage/
  
  build-staging:
    stage: build
    needs: [prepare]
    only:
      - schedules
    before_script:
      - apt-get update && apt-get install -y zip
    script:
      - npm run build:staging
      - zip -r build-staging.zip build
    # cache:
    #   paths:
    #     - build
    #   <<: *global_cache
    #   policy: push
    artifacts:
      paths:
        - build-staging.zip
  
  deploy-serv1:
    stage: deploy
    needs: [build-staging]
    tags: [linux]
    only:
      - schedules
    #   # - branches@gitlab-org/gitlab
    before_script:
      - apt-get update && apt-get install -y lftp
    script:
      # temporarily using 'verify-certificate no'
      # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
      # variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
      - lftp -e "set ssl:verify-certificate no; open myserver.org; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
    # environment:
    #   name: staging
    #   url: http://dev.myserver.org/projects/my-project/build
    when: manual
    allow_failure: true
  
  build-production:
    stage: build
    needs: [prepare]
    only:
      - schedules
    before_script:
      - apt-get update && apt-get install -y zip
    script:
      - npm run build:staging
      - zip -r build-prod.zip build
    # cache:
    #   paths:
    #     - build
    #   <<: *global_cache
    #   policy: push
    artifacts:
      paths:
        - build-prod.zip
  
  deploy-serv2:
    stage: deploy
    needs: [build-production]
    tags: [linux]
    only:
      - schedules
      # - master
    before_script:
      - apt-get update && apt-get install -y lftp
    script:
      - sh deploy-prod
    environment:
      name: production
      url: http://www.my-project.org
    when: manual
    allow_failure: true
  
  ```
  
  **Tags:** continuous-integration, yaml
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-24. Score: 3 | Accepted answer
  
  I highly recommend using [npm ci](https://docs.npmjs.com/cli/ci.html) instead of `npm install`. The slow part of installing npm modules is figuring out everything that needs to be installed. If you commit your package-lock.json and use `npm ci`, it skips that part entirely and just looks at your lockfile to download what's needed. It's significantly faster.
  
  > **Comments:**
  > **username_0**: What are your thoughts on npm install with `--prefer-offline` ? I was using npm ci but switched to npm i with the usage of prefer offline. As of yet I don't have an opinion on either - other that npm ci FEELS better due do the use of the keyword `ci`  
  > **username_1**: I've never used `--prefer-offline`. In fact I didn't know it existed before your comment. I think the main benefit of `npm ci` is the fact that you know exactly what will be installed and there are no changes made to package.json. It also deletes node_modules before installing packages so you know it's a clean slate every time.  I've not yet seen issues with `npm ci` slowness, but you could use `--prefer-offline` with `npm ci`. Here's an example from the gitlab docs:  https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies  
  > **username_0**: Funny you should mention that gitlab doc - those docs are where I pulled most my current implementation from. It seems like the issue now is that compiling / extracting the cache takes longer than fetching all the npm packages.  
  > **username_0**: Circling back - ultimately I deemed my `prepare` job to be doing double work. I took `prepare` out which cut my workload in half. I don't know I'd call that optimized but it's certainly much better.  
  > **username_1**: Ahh, yes. I don't think I noticed you were doing both. And optimization is relative, so yes, it's optimized. :D  
  
  ---
  
  