Answers: <p>I agree with <a href="https://devops.stackexchange.com/users/17225/omri-spector">Omri</a> that you are heading the wrong way. I would recommend using a <a href="https://jenkins.io/doc/book/pipeline/shared-libraries/" rel="nofollow noreferrer">shared library</a> which implements the pipeline. You would then have a simple Jenkins file which looks all the same. See <a href="https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/" rel="nofollow noreferrer">https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/</a></p>
  
  <p>Let's say you have a shared library which contains a <code>vars/commonpipeline.groovy</code> which contains something like this:</p>
  
  <pre><code>def call(body) {
      // evaluate the body block, and collect configuration into the object
      pipelineParams= [:]
      body.resolveStrategy = Closure.DELEGATE_FIRST
      body.delegate = pipelineParams
      body()
  
      pipeline {
          agent {
              label pipelineParams.nodes
          }
          options {
              buildDiscarder(logRotator(artifactNumToKeepStr: '1', numToKeepStr: '10'))
              disableConcurrentBuilds()
              timestamps()
              timeout(time: pipelineParams.timeout, unit: 'MINUTES')
          }
  
          stages {
              stage('Build') {
                  steps {
                      //TO SOMETHING
                  }
              }
          }
      }
  }
  </code></pre>
  
  <p>Then you can have a <code>Jenkinsfile</code> in your dev project which loads the library and uses the common pipeline like this:</p>
  
  <pre><code>import org.apache.commons.io.FileUtils
  library identifier: 'shared-pipeline-library@master', retriever: modernSCM(
    [$class: 'GitSCMSource',
         remote: 'https://bitbucket/jenkins/hared-pipeline-library.git',
         credentialsId: 'bitbucket.service.user'
    ])
  
  commonpipeline {
      nodes     = 'BUILD' /* label of jenkins nodes*/
      timeout   = '60'
  }
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2019-09-25T05:55:35.037Z
Answers: 2020-05-04T05:20:11.117Z
Answers: 2020-05-04T05:20:11.117Z
Answers: papanito
Answers: <p>There may not be a way to make your idea work with submodules. When a Jenkins multibranch pipeline polls SCM for changes, it has no visibility into submodules. I verified this recently. The result in &quot;Scan Multibranch Pipeline Log&quot; was a <code>Communication error for url</code> response. The Jenkinsfile was not found, and so no build was triggered.</p>
  <p>The good news is that I discovered a straightforward way to achieve what we're after <em>without</em> using submodules. Have a look at the <a href="https://plugins.jenkins.io/remote-file/" rel="nofollow noreferrer">Remote Jenkinsfile Provider Plugin</a>.</p>
  <p>When a pipeline is polling a repository to trigger new builds, it looks for two things for each branch:</p>
  <ol>
  <li>The polled branch has new SCM changes since the last build (HEAD is different).</li>
  <li>A Jenkinsfile exists in repository X on branch Y at path Z.</li>
  </ol>
  <p>In a multibranch pipeline's Build Configuration section, normally only one Mode is available to tell Jenkins where to find the Jenkinsfile: &quot;by Jenkinsfile&quot;. This mode allows us to specify Z (path) only; X and Y are <em>implicitly</em> set to the repository and branch being polled.</p>
  <p>What the Remote Jenkinsfile Provider Plugin gives us is a new Build Configuration Mode: &quot;by Remote Jenkins File Plugin&quot;. This mode allows us to <em>explicitly</em> set the repository, branch, and path for the Jenkinsfile - X, Y, <em>and</em> Z. In your example, X would be <code>ci-repo</code>, but <em>any</em> Jenkinsfile in <em>any</em> repository is fair game. You could have <code>ci-repo</code> be a submodule of <code>dev-repo</code> if it's convenient, but no such relationship is required to make this work.</p>
  <p>If desired, a &quot;Match branches&quot; option uses the <em>polled branch name</em> for Y, looking for a <em>remote</em> Jenkinsfile on a branch of the same name. For example, while polling <code>dev-repo/feature-123</code>, it would look for a Jenkinsfile on <code>ci-repo/feature-123</code>.</p>
  <p>As long as the Jenkinsfile exists at the specified location, the second criteria for triggering new builds will be satisfied. You'll see something like this in the pipeline's polling logs:</p>
  <pre><code>Looking up DR/dev-repo for branches
  Checking branch master from DR/dev-repo
  No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
      Met criteria
  Scheduled build for branch: master
  </code></pre>
  <p>Instructions for how to set this up, with screenshots, can be found on <a href="https://plugins.jenkins.io/remote-file/" rel="nofollow noreferrer">the plugin's page</a>.</p>
  <p><strong>Caveat:</strong> Pushing changes to a <em>remote</em> Jenkinsfile (<code>ci-repo</code>) will <em>not</em> trigger any builds of the polling repository (<code>dev-repo</code>). If this behavior is desired, one solution would be to use webhooks. Another would be to create a pipeline that polls the <em>remote</em> Jenkinsfile's repository and builds other pipeline jobs as desired using the <a href="https://www.jenkins.io/doc/pipeline/steps/pipeline-build-step/" rel="nofollow noreferrer">build step</a>, like so:</p>
  <pre><code>build(job: &quot;dev-repo/${BRANCH_NAME}&quot;, wait: false)
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2020-10-07T20:41:36.993Z
Answers: 2020-10-07T20:41:36.993Z
Answers: Ben Amos
Answers: <p>I believe you are heading in the wrong direction.
  The way to implement what you want is to separate the "devops infrastructure logic" (placed in the 'ci-repo') from the pipeline trigger - a jenkinsfile inside the dev-repo, which includes the "devops infrastructure logic" as a tagged artifact.</p>
  
  <p>Pros:</p>
  
  <ul>
  <li>You have a central place to develop (and test) devops code</li>
  <li>When changes are backward compatible, you can update all pipelines by moving the tag</li>
  <li>When changes are NOT backward compatible, repos &amp; specific branches can gradually move to a new tag</li>
  <li>There is room for specific repos &amp; branches to create their unique process if they have to</li>
  </ul>
  
  <p>Cons:</p>
  
  <ul>
  <li>Yes, sometimes people will need to update Jenkinsfiles in multiple places as they are accepting a new version. I actually think it's good.</li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2019-09-23T09:22:58.543Z
Answers: 2019-09-23T09:22:58.543Z
Answers: Omri Spector
Answers: <p>We have the exact same problem, and wanted to run a multibranch pipeline using a Jenkinsfile located in a git submodule. Our setup is a bit different because the submodule contains also libraries that are shared amongst several projects. So this is more than a <em>ci-repo</em>, but the principle stay the same. We prefer this approach because it allows linking easily the version of the project to the version of the pipeline that was used at release (instead of manually pointing to some revision of a shared library).</p>
  <p>Our solution is actually a direct application of <a href="https://stackoverflow.com/questions/37800195/how-do-you-load-a-groovy-file-and-execute-it">https://stackoverflow.com/questions/37800195/how-do-you-load-a-groovy-file-and-execute-it</a>.</p>
  <p>Let's say we have following project tree:</p>
  <pre><code>build/
      Jenkinsfile
  src/
      ...
  common/           &lt;-- a git submodule
      src/
          ...
  </code></pre>
  <p>Our Jenkinsfile is something like</p>
  <pre class="lang-java prettyprint-override"><code>    // file: build/Jenkinsfile
  
      def doThings()
      {
          // ...
      }
  
      try {
          node {
              // ...
          }
          // ...
      }
      catch(e) {
          // ...
      }
      // ...
  </code></pre>
  <p>So basically some utility functions and some pipeline stages (here in a try-catch for instance).</p>
  <p>Two actions are required to move this file to our <code>common</code> submodule:</p>
  <ul>
  <li>Move the file to submodule and adapt it slightly.</li>
  <li>Replace the original jenkinsfile with a small boilerplate version.</li>
  </ul>
  <p>Our moved Jenkinsfile becomes</p>
  <pre class="lang-java prettyprint-override"><code>    // file: common/build/Jenkinsfile
  
      def doThings()
      {
          // ...
      }
  
      def call() {
          try {
              // ...
          }
          // ...
      }
      return this
  </code></pre>
  <p>So basically we surround the pipeline in a call function, and don't forget to return <code>this</code> at the end of the file.</p>
  <p>The original Jenkinsfile is then replaced with this small boilerplate code as explained in the original post:</p>
  <pre class="lang-java prettyprint-override"><code>// file: build/Jenkinsfile
  
  def myPipeline
  node {
      checkout scm // mandatory
      myPipeline = load &quot;common/build/Jenkinsfile&quot;
  }
  myPipeline()
  </code></pre>
  <p>Of course this can be tuned further to pass parameters, use several jenkinsfile, etc. For instance reusing ideas from the <a href="https://www.jenkins.io/doc/book/pipeline/shared-libraries/" rel="nofollow noreferrer">documentation</a>:</p>
  <pre class="lang-java prettyprint-override"><code>def call(Map params)
  {
      println &quot;Building for project &quot; + params.projectName
      // ...
  }
  </code></pre>
  <pre class="lang-java prettyprint-override"><code>// ...
  myPipeline(projectName: &quot;MyProject&quot;)
  </code></pre>
  <p>Also note that this checkout the whole project, which might be inefficient. A better approach would be to checkout only the submodule that contains the Jenkinsfile.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-10-14T09:43:24.223Z
Answers: 2022-10-14T09:43:24.223Z
Answers: fuujuhi
Body: <p><strong>Background:</strong>
  The nature of our project's source involves multiple long-living branches, each one has its own Jenkins job (manually created).</p>
  
  <p><strong>Multibranch limitation:</strong>
  Unfortunately we do not use a Multibranch Pipeline to organize the project, because when (for example) a change is needed in the Pipeline, it would need to be added (manually) to each of the long-living branches' Jenkinsfiles.</p>
  
  <p><strong>Current implementation:</strong>
  We store the Jenkinsfile in a separate repository (let's call it <em>ci-repo</em>), which is added as a submodule to the project's repository (let's call that <em>dev-repo</em>).</p>
  
  <p><strong>What still hurts:</strong>
  We need to manually monitor branches of the <em>dev-repo</em> and manage job creation.
  Multibranch pipelines support submodule checkout for source files, but branch discovery requires presence of a Jenkinsfile under the parent repository (in our case <em>dev-repo</em>).</p>
  
  <p><strong>The question:</strong>
  Jenkinsfile from git-submodule of Multibranch Pipeline - Is there a way to do this?
  Perhaps using an initial/mock Jenkinsfile in the <em>dev-repo</em>, with one stage in which is calls the Jenkinsfile from the submodule?
  Can a Jenkinsfile be over-written or replaced somehow during runtime?</p>
  
Comments: CC BY-SA 4.0
Comments: 2020-03-20T16:38:14.253Z
Comments: I have a similar problem: multiple repositories have the same steps to execute build and test. We created a shared library with all the functions (such as test, build, deploy), but again the basic logic of the JenkinsFile must be stored multiple times in all the repositories. Did you a find a solution?
Comments: user1382278
Tags: <jenkins-pipeline><git><jenkinsfile>
Title: Is there a way to use a Jenkinsfile from a git-submodule in a multibranch pipeline?
UserDisplayName: maze
ThreadText: # Is there a way to use a Jenkinsfile from a git-submodule in a multibranch pipeline?
  
  (Asked by: username_0 on 2019-09-23)
  
  **Background:**
  The nature of our project's source involves multiple long-living branches, each one has its own Jenkins job (manually created).
  
  **Multibranch limitation:**
  Unfortunately we do not use a Multibranch Pipeline to organize the project, because when (for example) a change is needed in the Pipeline, it would need to be added (manually) to each of the long-living branches' Jenkinsfiles.
  
  **Current implementation:**
  We store the Jenkinsfile in a separate repository (let's call it *ci-repo*), which is added as a submodule to the project's repository (let's call that *dev-repo*).
  
  **What still hurts:**
  We need to manually monitor branches of the *dev-repo* and manage job creation.
  Multibranch pipelines support submodule checkout for source files, but branch discovery requires presence of a Jenkinsfile under the parent repository (in our case *dev-repo*).
  
  **The question:**
  Jenkinsfile from git-submodule of Multibranch Pipeline - Is there a way to do this?
  Perhaps using an initial/mock Jenkinsfile in the *dev-repo*, with one stage in which is calls the Jenkinsfile from the submodule?
  Can a Jenkinsfile be over-written or replaced somehow during runtime?
  
  **Tags:** jenkins-pipeline, git, jenkinsfile
  
  > **Comments:**
  > **username_1**: I have a similar problem: multiple repositories have the same steps to execute build and test. We created a shared library with all the functions (such as test, build, deploy), but again the basic logic of the JenkinsFile must be stored multiple times in all the repositories. Did you a find a solution?  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2019-09-25. Score: 4
  
  I agree with [username_4](https://devops.stackexchange.com/users/17225/username_4) that you are heading the wrong way. I would recommend using a [shared library](https://jenkins.io/doc/book/pipeline/shared-libraries/) which implements the pipeline. You would then have a simple Jenkins file which looks all the same. See <https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/>
  
  Let's say you have a shared library which contains a `vars/commonpipeline.groovy` which contains something like this:
  
  ```
  def call(body) {
      // evaluate the body block, and collect configuration into the object
      pipelineParams= [:]
      body.resolveStrategy = Closure.DELEGATE_FIRST
      body.delegate = pipelineParams
      body()
  
      pipeline {
          agent {
              label pipelineParams.nodes
          }
          options {
              buildDiscarder(logRotator(artifactNumToKeepStr: '1', numToKeepStr: '10'))
              disableConcurrentBuilds()
              timestamps()
              timeout(time: pipelineParams.timeout, unit: 'MINUTES')
          }
  
          stages {
              stage('Build') {
                  steps {
                      //TO SOMETHING
                  }
              }
          }
      }
  }
  
  ```
  
  Then you can have a `Jenkinsfile` in your dev project which loads the library and uses the common pipeline like this:
  
  ```
  import org.apache.commons.io.FileUtils
  library identifier: 'shared-pipeline-library@master', retriever: modernSCM(
    [$class: 'GitSCMSource',
         remote: 'https://bitbucket/jenkins/hared-pipeline-library.git',
         credentialsId: 'bitbucket.service.user'
    ])
  
  commonpipeline {
      nodes     = 'BUILD' /* label of jenkins nodes*/
      timeout   = '60'
  }
  
  ```
  
  
  ---
  
  ### Answer by username_3 on 2020-10-07. Score: 2
  
  There may not be a way to make your idea work with submodules. When a Jenkins multibranch pipeline polls SCM for changes, it has no visibility into submodules. I verified this recently. The result in "Scan Multibranch Pipeline Log" was a `Communication error for url` response. The Jenkinsfile was not found, and so no build was triggered.
  
  The good news is that I discovered a straightforward way to achieve what we're after *without* using submodules. Have a look at the [Remote Jenkinsfile Provider Plugin](https://plugins.jenkins.io/remote-file/).
  
  When a pipeline is polling a repository to trigger new builds, it looks for two things for each branch:
  
  1. The polled branch has new SCM changes since the last build (HEAD is different).
  2. A Jenkinsfile exists in repository X on branch Y at path Z.
  
  In a multibranch pipeline's Build Configuration section, normally only one Mode is available to tell Jenkins where to find the Jenkinsfile: "by Jenkinsfile". This mode allows us to specify Z (path) only; X and Y are *implicitly* set to the repository and branch being polled.
  
  What the Remote Jenkinsfile Provider Plugin gives us is a new Build Configuration Mode: "by Remote Jenkins File Plugin". This mode allows us to *explicitly* set the repository, branch, and path for the Jenkinsfile - X, Y, *and* Z. In your example, X would be `ci-repo`, but *any* Jenkinsfile in *any* repository is fair game. You could have `ci-repo` be a submodule of `dev-repo` if it's convenient, but no such relationship is required to make this work.
  
  If desired, a "Match branches" option uses the *polled branch name* for Y, looking for a *remote* Jenkinsfile on a branch of the same name. For example, while polling `dev-repo/feature-123`, it would look for a Jenkinsfile on `ci-repo/feature-123`.
  
  As long as the Jenkinsfile exists at the specified location, the second criteria for triggering new builds will be satisfied. You'll see something like this in the pipeline's polling logs:
  
  ```
  Looking up DR/dev-repo for branches
  Checking branch master from DR/dev-repo
  No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
      Met criteria
  Scheduled build for branch: master
  
  ```
  
  Instructions for how to set this up, with screenshots, can be found on [the plugin's page](https://plugins.jenkins.io/remote-file/).
  
  **Caveat:** Pushing changes to a *remote* Jenkinsfile (`ci-repo`) will *not* trigger any builds of the polling repository (`dev-repo`). If this behavior is desired, one solution would be to use webhooks. Another would be to create a pipeline that polls the *remote* Jenkinsfile's repository and builds other pipeline jobs as desired using the [build step](https://www.jenkins.io/doc/pipeline/steps/pipeline-build-step/), like so:
  
  ```
  build(job: "dev-repo/${BRANCH_NAME}", wait: false)
  
  ```
  
  
  ---
  
  ### Answer by username_4 on 2019-09-23. Score: 1
  
  I believe you are heading in the wrong direction.
  The way to implement what you want is to separate the "devops infrastructure logic" (placed in the 'ci-repo') from the pipeline trigger - a jenkinsfile inside the dev-repo, which includes the "devops infrastructure logic" as a tagged artifact.
  
  Pros:
  
  * You have a central place to develop (and test) devops code
  * When changes are backward compatible, you can update all pipelines by moving the tag
  * When changes are NOT backward compatible, repos & specific branches can gradually move to a new tag
  * There is room for specific repos & branches to create their unique process if they have to
  
  Cons:
  
  * Yes, sometimes people will need to update Jenkinsfiles in multiple places as they are accepting a new version. I actually think it's good.
  
  
  ---
  
  ### Answer by username_5 on 2022-10-14. Score: 0
  
  We have the exact same problem, and wanted to run a multibranch pipeline using a Jenkinsfile located in a git submodule. Our setup is a bit different because the submodule contains also libraries that are shared amongst several projects. So this is more than a *ci-repo*, but the principle stay the same. We prefer this approach because it allows linking easily the version of the project to the version of the pipeline that was used at release (instead of manually pointing to some revision of a shared library).
  
  Our solution is actually a direct application of <https://stackoverflow.com/questions/37800195/how-do-you-load-a-groovy-file-and-execute-it>.
  
  Let's say we have following project tree:
  
  ```
  build/
      Jenkinsfile
  src/
      ...
  common/           <-- a git submodule
      src/
          ...
  
  ```
  
  Our Jenkinsfile is something like
  
  ```java
      // file: build/Jenkinsfile
  
      def doThings()
      {
          // ...
      }
  
      try {
          node {
              // ...
          }
          // ...
      }
      catch(e) {
          // ...
      }
      // ...
  
  ```
  
  So basically some utility functions and some pipeline stages (here in a try-catch for instance).
  
  Two actions are required to move this file to our `common` submodule:
  
  * Move the file to submodule and adapt it slightly.
  * Replace the original jenkinsfile with a small boilerplate version.
  
  Our moved Jenkinsfile becomes
  
  ```java
      // file: common/build/Jenkinsfile
  
      def doThings()
      {
          // ...
      }
  
      def call() {
          try {
              // ...
          }
          // ...
      }
      return this
  
  ```
  
  So basically we surround the pipeline in a call function, and don't forget to return `this` at the end of the file.
  
  The original Jenkinsfile is then replaced with this small boilerplate code as explained in the original post:
  
  ```java
  // file: build/Jenkinsfile
  
  def myPipeline
  node {
      checkout scm // mandatory
      myPipeline = load "common/build/Jenkinsfile"
  }
  myPipeline()
  
  ```
  
  Of course this can be tuned further to pass parameters, use several jenkinsfile, etc. For instance reusing ideas from the [documentation](https://www.jenkins.io/doc/book/pipeline/shared-libraries/):
  
  ```java
  def call(Map params)
  {
      println "Building for project " + params.projectName
      // ...
  }
  
  ```
  
  ```java
  // ...
  myPipeline(projectName: "MyProject")
  
  ```
  
  Also note that this checkout the whole project, which might be inefficient. A better approach would be to checkout only the submodule that contains the Jenkinsfile.
  
  
  ---
  
  