Answers: <blockquote>
  <p>If this is not possible is there any other way without duplicating the 100GB for each jenkins job?</p>
  </blockquote>
  <p>To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path.  Then, in your Jenkins job, you can cd to that directory with <a href="https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#dir-change-current-directory" rel="nofollow noreferrer">the dir step</a>, update the repo, and checkout the branch you want, e.g.:</p>
  <pre><code>dir('/path/to/repo') {
    sh('git fetch --all')
    sh('git checkout origin/lineage-16.0')
    // your build steps here
  }
  </code></pre>
  <p>However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption.  It sounds like running jobs in parallel was one of your requirements, so this may be a no-go.</p>
  <p>You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-03-12T00:40:03.073Z
Answers: 2021-03-12T00:40:03.073Z
Answers: jayhendren
Answers: <p>Why is your &quot;source repo&quot; 100GB?</p>
  <p>Admittedly, the article is several years old, but MS reported <a href="https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/" rel="nofollow noreferrer">the windows code base repo was the largest known repo at the time and was 300GB</a>.</p>
  <p>Is your <strong>source</strong> really that big?  Your Git repo should NOT contain derived artifacts (.class, .jar/war, .o, .dll, .exe, etc), nor 3rd-party libraries. There are other tools for that.   You can use a <a href="https://git-scm.com/docs/gitignore" rel="nofollow noreferrer">.gitignore</a> file to exclude these file types from being checked into your repo.</p>
  <p>If you check in a binary file, Git will generally store a complete copy (generally non-compressible) of every copy you check in. I have seen developers check-in not just 3rd-party OSS jars in multiple locations in the same repo and same jars across multiple repos, resulting in massive duplication and wasted space, but then keep checking new versions. I've seen both parts of the Weblogic installer ( 2x 600 MB+) checked-in. And found one which was continuously checking in the Docker image he built which included 3party installers, making his repo almost 1GB larger every check-in!</p>
  <p>You don't mention language, but the standard tools for Java libraries has been <a href="https://www.sonatype.com/nexus-repository-oss" rel="nofollow noreferrer">Nexus</a> and <a href="https://jfrog.com/artifactory/" rel="nofollow noreferrer">Artifactory</a>. They now handle as extensive range of language artifact types as well.</p>
  <p>These tool also create local caches of the artifacts they retrieve so you no longer have to go to the remote host across the network if they have not changed.</p>
  <p><a href="https://git-lfs.github.com/" rel="nofollow noreferrer">Git-LFS</a> is an excellent extension to handle large, non-derived binary files (audio samples, videos, datasets, and graphics).</p>
  <p>Look at using <a href="https://rtyley.github.io/bfg-repo-cleaner/" rel="nofollow noreferrer">BFG</a> to remove unwanted artifacts from history and compress the repo.</p>
  <p>As documented in the &quot;largest repo ever&quot; article, MS have contributed the &quot;<a href="https://vfsforgit.org/" rel="nofollow noreferrer">VFS for Git</a>&quot; to deal with really large repos if that really is the only solution available to you.</p>
  <p>This is not an extensive how-to, just advice, &quot;I think you're doing it wrong and that's why it's a PITA&quot;, plus some pointers to tools. You need to review your entire approach. That may include other tools like <a href="https://www.ansible.com/" rel="nofollow noreferrer">Ansible</a> or <a href="https://puppet.com/" rel="nofollow noreferrer">Puppet</a> to manage your infra configuration as well.</p>
  <p><strong>Summary:</strong> Manage your infrastructure as code and keep only source code in your source code repositories. Or just get a <a href="https://www.sciencedaily.com/releases/2020/05/200522095504.htm" rel="nofollow noreferrer">faster network</a>.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-08-31T09:02:45.900Z
Answers: It's not my repository. It's the one for AOSP and LineageOS and other Custom ROMs. I don't have any access to it and can do nothing about their size. I need to mirror and re-use them.
Answers: dtrunk
Answers: CC BY-SA 4.0
Answers: 2020-08-31T09:48:42.933Z
Answers: Then I'd suggest the best place for guidance would be their respective support sites. This ia not a Jenkins issue. I'm reasonably sure you can't be the only user struggling with such larhe repos. They must have guidance ams strategy. The [AOSP site](https://source.android.com/setup/build/downloading) tallks about "you can specify `--partial-clone` when performing repo init which will make use of Git's partial clone capability, which only downloads Git objects when needed instead of downloading everything"
Answers: Ian W
Answers: CC BY-SA 4.0
Answers: 2020-08-30T23:10:44.343Z
Answers: 2020-08-31T03:12:13.783Z
Answers: 2020-08-31T03:12:13.783Z
Answers: Ian W
Body: <p>I would like to setup a Jenkins cluster where the slaves are building an Android ROM with some parameters like branch (e.g. <code>lineage-16.0</code>) and some optional patches. The whole repository will consume around 100GB.</p>
  <p>Let's say i want to build branch <code>lineage-16.0</code> with patch-A and patch-B and in parallel build branch <code>lineage-16.0</code> with patch-A only and in parallel build branch <code>lineage-17.1</code> with no patches and so on.</p>
  <p>I just read about the <code>--reference</code> option in <code>git</code> (which can also be used in <code>repo</code>).
  Is it possible to use it this way or will changing the branch and applying patches modify the reference clone as well as it only uses pointers?</p>
  <p>I need to save disk space and network traffic somehow.
  If this is not possible is there any other way without duplicating the 100GB for each jenkins job?
  I'm also open for other free tools than Jenkins.</p>
  
Tags: <jenkins><jenkins-pipeline><git>
Title: Jenkins: Re-use repository with reference option and do changes on it
UserDisplayName: dtrunk
ThreadText: # Jenkins: Re-use repository with reference option and do changes on it
  
  (Asked by: username_0 on 2020-08-29)
  
  I would like to setup a Jenkins cluster where the slaves are building an Android ROM with some parameters like branch (e.g. `lineage-16.0`) and some optional patches. The whole repository will consume around 100GB.
  
  Let's say i want to build branch `lineage-16.0` with patch-A and patch-B and in parallel build branch `lineage-16.0` with patch-A only and in parallel build branch `lineage-17.1` with no patches and so on.
  
  I just read about the `--reference` option in `git` (which can also be used in `repo`).
  Is it possible to use it this way or will changing the branch and applying patches modify the reference clone as well as it only uses pointers?
  
  I need to save disk space and network traffic somehow.
  If this is not possible is there any other way without duplicating the 100GB for each jenkins job?
  I'm also open for other free tools than Jenkins.
  
  **Tags:** jenkins, jenkins-pipeline, git
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-03-12. Score: 0
  
  > If this is not possible is there any other way without duplicating the 100GB for each jenkins job?
  
  To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path. Then, in your Jenkins job, you can cd to that directory with [the dir step](https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#dir-change-current-directory), update the repo, and checkout the branch you want, e.g.:
  
  ```
  dir('/path/to/repo') {
    sh('git fetch --all')
    sh('git checkout origin/lineage-16.0')
    // your build steps here
  }
  
  ```
  
  However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption. It sounds like running jobs in parallel was one of your requirements, so this may be a no-go.
  
  You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.
  
  
  ---
  
  ### Answer by username_2 on 2020-08-30. Score: -1
  
  Why is your "source repo" 100GB?
  
  Admittedly, the article is several years old, but MS reported [the windows code base repo was the largest known repo at the time and was 300GB](https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/).
  
  Is your **source** really that big? Your Git repo should NOT contain derived artifacts (.class, .jar/war, .o, .dll, .exe, etc), nor 3rd-party libraries. There are other tools for that. You can use a [.gitignore](https://git-scm.com/docs/gitignore) file to exclude these file types from being checked into your repo.
  
  If you check in a binary file, Git will generally store a complete copy (generally non-compressible) of every copy you check in. I have seen developers check-in not just 3rd-party OSS jars in multiple locations in the same repo and same jars across multiple repos, resulting in massive duplication and wasted space, but then keep checking new versions. I've seen both parts of the Weblogic installer ( 2x 600 MB+) checked-in. And found one which was continuously checking in the Docker image he built which included 3party installers, making his repo almost 1GB larger every check-in!
  
  You don't mention language, but the standard tools for Java libraries has been [Nexus](https://www.sonatype.com/nexus-repository-oss) and [Artifactory](https://jfrog.com/artifactory/). They now handle as extensive range of language artifact types as well.
  
  These tool also create local caches of the artifacts they retrieve so you no longer have to go to the remote host across the network if they have not changed.
  
  [Git-LFS](https://git-lfs.github.com/) is an excellent extension to handle large, non-derived binary files (audio samples, videos, datasets, and graphics).
  
  Look at using [BFG](https://rtyley.github.io/bfg-repo-cleaner/) to remove unwanted artifacts from history and compress the repo.
  
  As documented in the "largest repo ever" article, MS have contributed the "[VFS for Git](https://vfsforgit.org/)" to deal with really large repos if that really is the only solution available to you.
  
  This is not an extensive how-to, just advice, "I think you're doing it wrong and that's why it's a PITA", plus some pointers to tools. You need to review your entire approach. That may include other tools like [Ansible](https://www.ansible.com/) or [Puppet](https://puppet.com/) to manage your infra configuration as well.
  
  **Summary:** Manage your infrastructure as code and keep only source code in your source code repositories. Or just get a [faster network](https://www.sciencedaily.com/releases/2020/05/200522095504.htm).
  
  > **Comments:**
  > **username_0**: It's not my repository. It's the one for AOSP and LineageOS and other Custom ROMs. I don't have any access to it and can do nothing about their size. I need to mirror and re-use them.  
  > **username_2**: Then I'd suggest the best place for guidance would be their respective support sites. This ia not a Jenkins issue. I'm reasonably sure you can't be the only user struggling with such larhe repos. They must have guidance ams strategy. The [AOSP site](https://source.android.com/setup/build/downloading) tallks about "you can specify `--partial-clone` when performing repo init which will make use of Git's partial clone capability, which only downloads Git objects when needed instead of downloading everything"  
  
  ---
  
  