Answers: <p>Yes, you should definitively run both parts in parallel if you have the resources. This means that your job will run as fast as your slowest running part. This puts emphasis on optimizing your slowest running part. It might be something as simple as running some processes of part 1 ahead of time before part 2 even begins. </p>
  
Answers: CC BY-SA 3.0
Answers: 2017-11-17T20:45:35.733Z
Answers: 2017-11-17T20:45:35.733Z
Answers: Preston Martin
Answers: <p>The critical question is - can you teach your job #2 to patiently wait for job #1 to complete when it gets to that dependency point and appropriately proceed or bail depending on job #1 result?</p>
  
  <p>If you can't, then the answer is no - you'll have more trouble dealing with #2 failures due to incomplete job #1, too much noise in the process.</p>
  
  <p>If you can (and if you have enough resources) - yes, by all means - your process will be faster. You <em>might</em> not even need special support from the CI system for such dependencies, it could be completely oblivious, it would simply see the 2 jobs as parallel.</p>
  
  <p>A middle-ground possibility, if your CI systems supports it, would be to stagger the jobs: start job #2 with a delay sufficient to bring the chances of job #1 completing before job #2 needs it in a much more comfortable range (say 99%). The delay would be plain eyeballed or empirically determined, either fixed or, in a more advanced approach, automatically derived from historic measurements/stats (if available).</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-11-16T21:26:20.493Z
Answers: 2017-11-16T21:26:20.493Z
Answers: Dan Cornilescu
Body: <p>Imagine you have resources for a job which has part 1 and part 2.</p>
  
  <p>Part 2 requires part 1, but both take considerable time.</p>
  
  <p>And, part 2 requires part 1 only at the end.</p>
  
  <p>Would you start them both in parallel and let part 2 somehow wait for part 1 getting ready? (chances are it 50/50 it will be ready by then)</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-11-20T13:17:31.010Z
Comments: Can Part 2 be split into two?  (i.e. can the dependent part be broken off into a Part 3, leaving Part 2 to be independent?)
Comments: Oliver Charlesworth
Tags: <continuous-integration><automation>
Title: Parallelizing dependent CI subjobs
UserDisplayName: Ta Mu
ThreadText: # Parallelizing dependent CI subjobs
  
  (Asked by: username_0 on 2017-11-16)
  
  Imagine you have resources for a job which has part 1 and part 2.
  
  Part 2 requires part 1, but both take considerable time.
  
  And, part 2 requires part 1 only at the end.
  
  Would you start them both in parallel and let part 2 somehow wait for part 1 getting ready? (chances are it 50/50 it will be ready by then)
  
  **Tags:** continuous-integration, automation
  
  > **Comments:**
  > **username_1**: Can Part 2 be split into two?  (i.e. can the dependent part be broken off into a Part 3, leaving Part 2 to be independent?)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-11-16. Score: 4 | Accepted answer
  
  The critical question is - can you teach your job #2 to patiently wait for job #1 to complete when it gets to that dependency point and appropriately proceed or bail depending on job #1 result?
  
  If you can't, then the answer is no - you'll have more trouble dealing with #2 failures due to incomplete job #1, too much noise in the process.
  
  If you can (and if you have enough resources) - yes, by all means - your process will be faster. You *might* not even need special support from the CI system for such dependencies, it could be completely oblivious, it would simply see the 2 jobs as parallel.
  
  A middle-ground possibility, if your CI systems supports it, would be to stagger the jobs: start job #2 with a delay sufficient to bring the chances of job #1 completing before job #2 needs it in a much more comfortable range (say 99%). The delay would be plain eyeballed or empirically determined, either fixed or, in a more advanced approach, automatically derived from historic measurements/stats (if available).
  
  
  ---
  
  ### Answer by username_2 on 2017-11-17. Score: 1
  
  Yes, you should definitively run both parts in parallel if you have the resources. This means that your job will run as fast as your slowest running part. This puts emphasis on optimizing your slowest running part. It might be something as simple as running some processes of part 1 ahead of time before part 2 even begins.
  
  
  ---
  
  