Answers: <p>Yes, but it's by preventing the things you mention in your question as much as possible. 
  You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.</p>
  
  <p>Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort.
  This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible.
  How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous <a href="https://en.wikipedia.org/wiki/DLL_Hell" rel="nofollow noreferrer">DLL Hell</a>.
  Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor) </p>
  
  <p>If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.</p>
  
  <p>When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-03T10:44:59.350Z
Answers: 2017-03-03T13:33:14.187Z
Answers: 2017-03-03T13:33:14.187Z
Answers: THelper
Answers: <blockquote>
    <p>As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears</p>
  </blockquote>
  
  <p>While the first part of the sentence is probably statistically correct I disagree with the second.</p>
  
  <p>CI and integration branches cannot defeat GIGO (garbage in garbage out) If the developers has no discipline then you will end up in a halted branch, but integration branch tests should only act as the lat barrier, not the first.</p>
  
  <p>Developers should use proper coding standards, code reviews, local unit tests and possibly pre-integration branch tests to make collisions less likely.</p>
  
  <p>I am working daily on such a system with hundreds of contributors and surprisingly it's uncommon to see  it hangs.</p>
  
  <p>I don't have enough data but it might be true that cost is high, on the other hand it's difficult to judge the alternative cost of late integration and bugs.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-03T13:33:52.950Z
Answers: 2017-03-03T13:33:52.950Z
Answers: Rsf
Answers: <p>Another approach is to switch to a different, non-traditional CI system, capable of <strong>preventing regressions</strong> by identifying and rejecting the offending candidate changes prior to their commit rather than detecting regressions and waiting for recovery while development on the branch is blocked. I like to call such system a <a href="https://www.apartsw.com/insights/2016/11/25/traditional_vs_non_blocking_ci.html" rel="nofollow noreferrer">non-blocking CI system</a>.</p>
  
  <p>Simply bumping up the developer-executed pre-commit verification requirements in an attempt to reduce branch regressions is not sufficient for such guarantee. It may actually cause a regression rate <em>increase</em>: longer verification times => higher number of changes simultaneously verified in isolation (thus not accounting for each-other) => higher risk of functional conflicts and interference. I'm explaining this in a bit more details in the <a href="https://www.apartsw.com/insights/2016/11/16/pre_commit_verification_myths.html#extensible" rel="nofollow noreferrer">Extending the pre-commit verification coverage should increase our branch stability</a> myth.</p>
  
  <p>The CI tool also needs to be capable of operating at scale, of course. And such tools exist, see <a href="https://devops.stackexchange.com/questions/268/is-there-a-ci-tool-that-guarantees-no-regressions-in-the-branch-quality-level">Is there a CI tool that guarantees no regressions in the branch quality level?</a></p>
  
  <p><sup>Disclosure: I'm affiliated to the company mentioned in the above-links.</sup></p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-06T23:32:51.653Z
Answers: You mind if I rework your "disclaimer" a bit (if you don't like it, just perform a rollback)?
Answers: Pierre.Vriens
Answers: CC BY-SA 3.0
Answers: 2017-03-06T23:51:54.327Z
Answers: Not at all, please do.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-07T00:15:50.307Z
Answers: Done ... IMO equivalent to your version, compliant to SE requirements (to add a disclosure), and a more appropriate (less disturbing) font. Feel free to rework or rollback if you want to.
Answers: Pierre.Vriens
Answers: CC BY-SA 3.0
Answers: 2017-03-04T00:59:05.453Z
Answers: 2017-03-07T00:12:32.890Z
Answers: 2017-04-13T13:01:29.297Z
Answers: Dan Cornilescu
Body: <p>Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.</p>
  
  <p>As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.</p>
  
  <p>Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment <em>might</em> help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.</p>
  
  <p>Scalability is, at best, dubious.</p>
  
  <p>Is there a method for making such large projects actually scale?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-10T16:06:25.400Z
Comments: This question is, to me, like quite a few other questions that make me think like "why is this even a question". Not that it is a bad question, but rather something that in the world I'm coming from (mainframes, I bet you know by now ...) is not really a question anymore. That's what over there we're doing for at least a decade or so already, and IMO it works like a charm. How otherwise would it be possible to keep a banking system going. I just don't know if it makes sense to post an answer looking at it with mainframe glasses. What do you think?
Comments: Pierre.Vriens
Comments: CC BY-SA 3.0
Comments: 2017-03-10T16:32:46.587Z
Comments: The scale of the project here means the size and speed of the *changes* to the system being performed, not necessarily the size of the system itself (which would matter, for example, if the project would be re-writing the whole system or building it from scratch). And the context is a single branch, detaching on feature branches for days/weeks/months at a time is not continuous integration. Not saying that it doesn't happen at your place, only saying that the mainframe context itself doesn't imply large scale continuous integration. Think hundreds of commits per day into the same branch.
Comments: Dan Cornilescu
Comments: CC BY-SA 3.0
Comments: 2017-04-28T15:36:34.177Z
Comments: This question leaves me with so many more questions - how did you end up in this situation? Why are so many developers working on the same branch? Why would feature branches need their own QA environment? Why are changes being pushed to the integration branch that break the build, and why isn't that the problem being addressed rather than CI scaling? Are they not testing locally? Not merging from upstream before they merge to it?
Comments: Adrian
Comments: CC BY-SA 3.0
Comments: 2017-04-29T05:26:35.347Z
Comments: @Adrian Branches are evil - they bring the Integration Hell with them - risky, slow, expensive. Especially for large teams. Branches can easily diverge too much and make merging impractical. Lookup the CI advantages. Branching means working in isolation - canceling all those advantages and bringing back the complementary disadvantages. Yes, branches have been used to successfully ship sw for decades - there was no alternative. But now we have CI. Yet many still use branches - they're used to them, not even feeling the pain anymore or choosing to ignore it. Or hit scalability issues.
Comments: Dan Cornilescu
Comments: CC BY-SA 3.0
Comments: 2017-04-29T14:55:05.260Z
Comments: CI solves a completely different problem than branches, and used correctly, branches work with CI, not against it. Not using branches doesn't avoid integration hell either, it makes it worse: everyone working in the same branch has to merge every time they push.
Comments: Adrian
Comments: CC BY-SA 3.0
Comments: 2017-04-30T05:41:14.223Z
Comments: @Adrian Yes - the perfect merges: done by *exactly* the right person you want to be doing each and every one of them, with no delay and confirmed to be completed successfully or having problems at the very next CI cycle - just a regular change (even if technically it's a branch merge under the hood). This needs to be done anyways, going on a branch won't avoid it, it'll just delay it (while making it worse). **Note:** I should have specified - by branches I meant branches intended to be merged for integration. I'm OK with release branches (as long as they aren't merged with other branches).
Comments: Dan Cornilescu
Comments: CC BY-SA 3.0
Comments: 2017-04-30T14:12:24.680Z
Comments: Is that in contrast to branch merges? If your branch merges are being done by the wrong person at the wrong time and not validated in the next CI cycle, that might be part of the problem. Claiming "branches are evil" implies some bad experiences caused by poor practices and blamed on branches, like a carpenter hitting his thumb and declaring hammers are evil.
Comments: Adrian
Comments: CC BY-SA 3.0
Comments: 2017-04-30T22:37:53.643Z
Comments: @Adrian I can't be swayed to ever re-consider integration branches and I know and accept that some can't be bothered with CI (as in trunk based development). Let's agree to disagree and leave it at that - this is not the right place for such debate.
Comments: Dan Cornilescu
Tags: <continuous-integration><scalability><capacity-planning>
Title: How can continuous integration scale for very large projects/teams?
UserDisplayName: Dan Cornilescu
ThreadText: # How can continuous integration scale for very large projects/teams?
  
  (Asked by: username_0 on 2017-03-03)
  
  Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.
  
  As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.
  
  Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment *might* help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.
  
  Scalability is, at best, dubious.
  
  Is there a method for making such large projects actually scale?
  
  **Tags:** continuous-integration, scalability, capacity-planning
  
  > **Comments:**
  > **username_1**: This question is, to me, like quite a few other questions that make me think like "why is this even a question". Not that it is a bad question, but rather something that in the world I'm coming from (mainframes, I bet you know by now ...) is not really a question anymore. That's what over there we're doing for at least a decade or so already, and IMO it works like a charm. How otherwise would it be possible to keep a banking system going. I just don't know if it makes sense to post an answer looking at it with mainframe glasses. What do you think?  
  > **username_0**: The scale of the project here means the size and speed of the *changes* to the system being performed, not necessarily the size of the system itself (which would matter, for example, if the project would be re-writing the whole system or building it from scratch). And the context is a single branch, detaching on feature branches for days/weeks/months at a time is not continuous integration. Not saying that it doesn't happen at your place, only saying that the mainframe context itself doesn't imply large scale continuous integration. Think hundreds of commits per day into the same branch.  
  > **username_2**: This question leaves me with so many more questions - how did you end up in this situation? Why are so many developers working on the same branch? Why would feature branches need their own QA environment? Why are changes being pushed to the integration branch that break the build, and why isn't that the problem being addressed rather than CI scaling? Are they not testing locally? Not merging from upstream before they merge to it?  
  > **username_0**: @username_2 Branches are evil - they bring the Integration Hell with them - risky, slow, expensive. Especially for large teams. Branches can easily diverge too much and make merging impractical. Lookup the CI advantages. Branching means working in isolation - canceling all those advantages and bringing back the complementary disadvantages. Yes, branches have been used to successfully ship sw for decades - there was no alternative. But now we have CI. Yet many still use branches - they're used to them, not even feeling the pain anymore or choosing to ignore it. Or hit scalability issues.  
  > **username_2**: CI solves a completely different problem than branches, and used correctly, branches work with CI, not against it. Not using branches doesn't avoid integration hell either, it makes it worse: everyone working in the same branch has to merge every time they push.  
  > **username_0**: @username_2 Yes - the perfect merges: done by *exactly* the right person you want to be doing each and every one of them, with no delay and confirmed to be completed successfully or having problems at the very next CI cycle - just a regular change (even if technically it's a branch merge under the hood). This needs to be done anyways, going on a branch won't avoid it, it'll just delay it (while making it worse). **Note:** I should have specified - by branches I meant branches intended to be merged for integration. I'm OK with release branches (as long as they aren't merged with other branches).  
  > **username_2**: Is that in contrast to branch merges? If your branch merges are being done by the wrong person at the wrong time and not validated in the next CI cycle, that might be part of the problem. Claiming "branches are evil" implies some bad experiences caused by poor practices and blamed on branches, like a carpenter hitting his thumb and declaring hammers are evil.  
  > **username_0**: @username_2 I can't be swayed to ever re-consider integration branches and I know and accept that some can't be bothered with CI (as in trunk based development). Let's agree to disagree and leave it at that - this is not the right place for such debate.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-03-03. Score: 5
  
  Yes, but it's by preventing the things you mention in your question as much as possible.
  You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.
  
  Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort.
  This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible.
  How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous [DLL Hell](https://en.wikipedia.org/wiki/DLL_Hell).
  Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)
  
  If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.
  
  When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.
  
  
  ---
  
  ### Answer by username_4 on 2017-03-03. Score: 4
  
  > As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears
  
  While the first part of the sentence is probably statistically correct I disagree with the second.
  
  CI and integration branches cannot defeat GIGO (garbage in garbage out) If the developers has no discipline then you will end up in a halted branch, but integration branch tests should only act as the lat barrier, not the first.
  
  Developers should use proper coding standards, code reviews, local unit tests and possibly pre-integration branch tests to make collisions less likely.
  
  I am working daily on such a system with hundreds of contributors and surprisingly it's uncommon to see it hangs.
  
  I don't have enough data but it might be true that cost is high, on the other hand it's difficult to judge the alternative cost of late integration and bugs.
  
  
  ---
  
  ### Answer by username_0 on 2017-03-04. Score: 0
  
  Another approach is to switch to a different, non-traditional CI system, capable of **preventing regressions** by identifying and rejecting the offending candidate changes prior to their commit rather than detecting regressions and waiting for recovery while development on the branch is blocked. I like to call such system a [non-blocking CI system](https://www.apartsw.com/insights/2016/11/25/traditional_vs_non_blocking_ci.html).
  
  Simply bumping up the developer-executed pre-commit verification requirements in an attempt to reduce branch regressions is not sufficient for such guarantee. It may actually cause a regression rate *increase*: longer verification times => higher number of changes simultaneously verified in isolation (thus not accounting for each-other) => higher risk of functional conflicts and interference. I'm explaining this in a bit more details in the [Extending the pre-commit verification coverage should increase our branch stability](https://www.apartsw.com/insights/2016/11/16/pre_commit_verification_myths.html#extensible) myth.
  
  The CI tool also needs to be capable of operating at scale, of course. And such tools exist, see [Is there a CI tool that guarantees no regressions in the branch quality level?](https://devops.stackexchange.com/questions/268/is-there-a-ci-tool-that-guarantees-no-regressions-in-the-branch-quality-level)
  
  Disclosure: I'm affiliated to the company mentioned in the above-links.
  
  > **Comments:**
  > **username_1**: You mind if I rework your "disclaimer" a bit (if you don't like it, just perform a rollback)?  
  > **username_0**: Not at all, please do.  
  > **username_1**: Done ... IMO equivalent to your version, compliant to SE requirements (to add a disclosure), and a more appropriate (less disturbing) font. Feel free to rework or rollback if you want to.  
  
  ---
  
  