Answers: <p><a href="https://apartci.apartsw.com/" rel="nofollow noreferrer">ApartCI</a> is a CI system designed exactly to prevent regressions, thus guaranteeing flat or increasing branch quality level. Still in beta.</p>
  
  <p>It orchestrates centralized pre-commit verifications in such manner as to ensure that a change is committed only after it is verified, together with all other changes committed before it, to meet or exceed the latest branch quality level. </p>
  
  <p>This is the key difference compared to traditional developer-driven pre-commit verifications, often done in parallel , which leaves room for regressions caused by interfering changes which were never tested together.</p>
  
  <p>The tool is also <a href="https://apartci.apartsw.com/help_scale.html" rel="nofollow noreferrer">designed to easily scale</a> - capable of sustaining very high rates of incoming candidate changes and supporting 100s/1000s of developers working in the same integration branch.</p>
  
  <p><strong>Disclaimer:</strong> I'm the author of the tool and founder of the company offering it. Apologies for the ad.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-04-01T23:21:36.850Z
Answers: SnapCI did this too. RIP.
Answers: paul_h
Answers: CC BY-SA 3.0
Answers: 2017-04-02T03:31:02.890Z
Answers: @paul_h, unless I'm missing something SnapCI and its recommended replacement GoCD are both based on post-commit verifications (triggered by polling the SCM), so they're still monitoring-only. Except maybe for PR checks, but unless these checks are completely serialized they can only reduce the regression occurence rates, but not eliminate them completely.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-04-08T19:07:09.053Z
Answers: Dan, not polling no, hooks. And to a short-lived PR branch that is not yet merged into trunk/master - https://trunkbaseddevelopment.com/game-changers/#snap-ci-s-per-commit-speculative-mergeability-analysis-2013
Answers: paul_h
Answers: CC BY-SA 3.0
Answers: 2017-04-08T20:24:16.780Z
Answers: Nice idea, but it still is just monitoring: it indicates if the speculative merge would be OK or not. If it's not OK human intervention is needed. And doesn't scale well - it's still done on the local repo, not on the remote one, for each and every commit.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-04T11:39:08.327Z
Answers: It's good that you added the disclaimer, but personally I consider asking a question and self-answering it by promoting your own company or products to be a form of spam.
Answers: THelper
Answers: CC BY-SA 3.0
Answers: 2017-03-04T12:04:42.033Z
Answers: I've asked a question on meta whether this is allowed or not: http://meta.devops.stackexchange.com/q/59
Answers: THelper
Answers: CC BY-SA 3.0
Answers: 2017-03-03T23:44:48.730Z
Answers: 2017-03-03T23:44:48.730Z
Answers: Dan Cornilescu
Answers: <p>As far as I can tell, you're looking for a tool that will <strong>reject</strong> commits that break the build—a CI tool probably won't be able to prevent regressions by actually fixing your code, but it can stop you from adding bad code to the repository.</p>
  
  <p><a href="https://www.atlassian.com/continuous-delivery/git-hooks-continuous-integration" rel="nofollow noreferrer">Atlassian has a few interesting applications of Git hooks</a>:</p>
  
  <blockquote>
    <p>Server-side pre-receive hooks are an especially powerful compliment to continuous integration because they can prevent developers from pushing code to master, unless the code meets certain conditions – like elite ninja guardians, protecting it from bad code.</p>
  </blockquote>
  
  <p>If you're using Git, the hooks are <strong>very</strong> powerful (and there are similar hooks for <a href="http://svnbook.red-bean.com/nightly/en/svn.reposadmin.create.html" rel="nofollow noreferrer">SVN</a>, <a href="http://hgbook.red-bean.com/read/handling-repository-events-with-hooks.html" rel="nofollow noreferrer">Mercurial</a> and other version control systems), and you might find it useful to use them to run pre-commit checks.</p>
  
  <p>The Git documentation <a href="https://git-scm.com/book/en/v2/Customizing-Git-An-Example-Git-Enforced-Policy" rel="nofollow noreferrer">has a page on creating a hook to reject pushes if they don't meet a certain criteria</a> which could easily be adapted to this use-case.</p>
  
  <p>However, a lot of people would argue that rejecting commits <a href="https://softwareengineering.stackexchange.com/questions/294228/automatically-reverting-commits-that-fail-the-build">is a bad idea</a> on a <code>feature</code> branch—you'll just be wasting time fighting against your CI system when the build breaks for some reason, instead of actually fixing the bug. </p>
  
  <p>On the <code>master</code> branch, it could make sense to reject broken merges, because you might want to ensure it always builds. For a <code>feature</code> branch, you <strong>will</strong> inevitably break things, and since the code isn't going into production <strong>now</strong>, it makes more sense just to warn you than actually reject your commit altogether.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-03T13:39:13.593Z
Answers: Well, what good is a sw image which builds, but is DOA from testing prospective? Devs can't test their code changes even if they build, so they'd be just as blocked. So in general I'd extend commit rejection to anything that fails a minimum QA check, chosen by balancing 2 conflicting requirements: as high as possible to maximize the number of devs protected from blocking and as low as possible such that the QA verification delays don't slow the process down too much.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-04T04:41:36.170Z
Answers: An example of this might be the pull request model where you can pose certain restrictions on whether a pull request can be merged or not. For example, we (my company) uses Atlassian Bitbucket Server and there are options to require at least N number of approvals and X number of passing builds for the given branch before a pull request is allowed to be merged. This doesn't outright reject it. But prevents accidental merging when tests fail or other eyes have not seen the code yet.
Answers: Andy Shinn
Answers: CC BY-SA 3.0
Answers: 2017-03-04T11:19:37.607Z
Answers: @AndyShinn: Yeah, I find that quite useful—GitHub also offers [protected branches and checks on pull requests](https://help.github.com/articles/enabling-required-status-checks/), which are often helpful.
Answers: Aurora0001
Answers: CC BY-SA 3.0
Answers: 2017-03-04T19:18:40.957Z
Answers: One argument for allowing broken code in feature branches is it allows devs to push code they are working on to the repo, even if it's not quite ready yet. This can be useful for sharing code with others for early code/architecture reviews before things are ready to go, help debugging issues, or for someone who is leaving on vacation to put partially done work where others can get to it. For feature branches, I would only put things like linters and as pre-commit hooks.
Answers: bradym
Answers: CC BY-SA 3.0
Answers: 2017-03-03T10:13:25.227Z
Answers: 2017-03-03T10:13:25.227Z
Answers: 2017-04-13T12:45:57.413Z
Answers: Aurora0001
Answers: <p>No tool could possibly <em>guarantee</em> no regressions - that depends much more on your tests than the tool executing them. However, you can help to prevent regressions that will be caught from entering into the integration branch. You can do this with pre-commit hooks, but it's often easier with pull requests (which hopefully you're already using for peer code review).</p>
  
  <p>If a branch is up-to-date with its upstream (where the PR is merging to), and its tests pass, then they will still pass after the merge; the state of the target branch after the merge will match the state of the source branch before the merge.</p>
  
  <p>It's generally not particularly difficult (depending on the tools used) to indicate both whether the source branch in a PR is up-to-date with the target, and if it has a passing CI build. You can use this as a requirement (by policy, and/or enforced in software) for merging the pull request.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-04T03:57:37.760Z
Answers: "If a branch is up-to-date with its upstream (where the PR is merging to), and its tests pass, then they will still pass after the merge" - Why? A merge is a discontinuity, it brings unknowns. Like conflicts - the code might not even build until they're resolved. You **need** to run the tests and confirm that they pass for any branch merge. I'd say even for a fast-forward one, if you want to play it safe. See https://www.apartsw.com/insights/2016/11/16/pre_commit_verification_myths.html
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-04T04:16:48.060Z
Answers: Um, yes, such guarantee is possible, check my answer. Well, maybe I should clarify: by regression I mean results worse that the branch baseline results (and ignoring possibility of false positives, those need to be addressed as they can skew the baseline, derailing the whole thing and requiring human intervention). Intermittent false negatives are just a nuisance, slowing things down, but can be dealt with.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-04T04:30:31.807Z
Answers: You can't guarantee no regressions, you can only guarantee no detected regressions. If a change causes a regression not caught by a test, that's a regression that a CI tool cannot make any guarantees about. And while merging two sets of changes does bring unknowns, you can choose to do that in the feature branch (by merging the upstream in) before merging the other direction. If the source is up to date with the target, it's a simple merge (fast forward), and afterward the target state will be identical to the source state before the merge, hence if tests passed before, they'll pass after.
Answers: Adrian
Answers: CC BY-SA 3.0
Answers: 2017-03-04T04:46:57.070Z
Answers: Splitting hairs. The CI tool can be configured with a test to detect and thus protect against whichever regression you care about. I won't argue too much about the merges - my goal is to avoid them as much as possible, they're just trouble overall :)
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-04T16:49:13.927Z
Answers: My point is that it's not the CI tool offering that protection, it's you, by writing the tests. The CI tool can't provide any guarantees beyond the tests you provide.
Answers: Adrian
Answers: CC BY-SA 3.0
Answers: 2017-03-05T04:09:14.393Z
Answers: OK, I added a clarification to the question.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-04T03:25:40.230Z
Answers: 2017-03-04T03:25:40.230Z
Answers: Adrian
Answers: <p>Use GitLAB, you can set in project-settings to only allow a merge when the pipeline succeed, so can have a truly Continuous Integration, combine that with adding your QA to the list of merge approvals and with Dynamic Environments, you can have quality assurance before you merge to the master.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-09-17T09:28:12.827Z
Answers: That works but only if you do not allow a 2nd merge to start QA checks before the previous merge is completed, otherwise the 2nd merge won't see the 1st one, leaving room for regressions. In other words the merges (including their QA checks) must be completely serialized, which doesn't scale for large teams.
Answers: Dan Cornilescu
Answers: CC BY-SA 4.0
Answers: 2018-09-17T04:07:11.553Z
Answers: 2018-09-17T04:07:11.553Z
Answers: Danny Goossen
Answers: <p>True continuous integration tools (as opposed to just continuous testing) like Reitveld and Zuul can help, though they are only as good as the tests you write and code reviews you do. </p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-03T07:09:28.037Z
Answers: But Reitveld appears to be a collaborative code review tool, not a CI tool, am I missing something? This is what I looked at: https://github.com/rietveld-codereview/rietveld
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-03T07:16:57.190Z
Answers: Zuul appears to be indeed, related, I'll study it closer.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-03T07:37:17.490Z
Answers: It doesn't do testing but it does handle some aspects of branch management, acting as a gatekeeper system, which is the best bet to keep bad code from getting in by blocking the merge.
Answers: coderanger
Answers: CC BY-SA 3.0
Answers: 2017-03-03T14:13:40.827Z
Answers: I see what you mean. Well, it can help with the overall code quality, but by itself doesn't bring any guarantee. Two perfectly good changes that pass all QA verifications independently can still cause breakages when they meet in the branch.
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-03-03T06:57:53.743Z
Answers: 2017-03-03T06:57:53.743Z
Answers: coderanger
Body: <p>Traditionally CI systems only perform <strong>monitoring</strong> of the quality levels in an integration branch, by performing QA verifications on the codebase where the changes are already committed, watching for regressions and sending notifications for human intervention. </p>
  
  <p>But when these regressions are detected the branch has already been in trouble at least since the respective QA verification started and will remain in such state (or even get worse!) until all the culprits are identified, repairs for them committed and a new QA verification confirms the branch quality level was restored. The branch can be considered blocked for normal development during all this time.</p>
  
  <p>Is there a CI tool capable of actually <strong>preventing</strong> such regressions from happening, which would perform <em>pre-commit</em> QA verifications and allowing commits only when the codebase updated with the respective commits would be passing those pre-commit QA verifications as well, thus guaranteeing a minimum branch quality level?</p>
  
  <p><strong>Update:</strong> the assumption is that suitable automated QA verifications with appropriate coverage to be able to detect the respective regressions are available for invocation by the CI tool(s).</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-29T08:31:43.657Z
Comments: I keep wondering about the correct way to understand this question (and your own recommendation in your own answer). If I'd replace "monitoring" with "after the facts", and "preventing" with "prvent them from happening", would that more or less be the same question then? Also, maybe you can add some example scenario to explain the difference?
Comments: Pierre.Vriens
Comments: CC BY-SA 3.0
Comments: 2017-03-29T12:27:38.107Z
Comments: @Pierre.Vriens Is this any better?
Comments: Dan Cornilescu
Tags: <continuous-integration><software-recommendation>
Title: Is there a CI tool that guarantees no regressions in the branch quality level?
UserDisplayName: Dan Cornilescu
ThreadText: # Is there a CI tool that guarantees no regressions in the branch quality level?
  
  (Asked by: username_0 on 2017-03-03)
  
  Traditionally CI systems only perform **monitoring** of the quality levels in an integration branch, by performing QA verifications on the codebase where the changes are already committed, watching for regressions and sending notifications for human intervention.
  
  But when these regressions are detected the branch has already been in trouble at least since the respective QA verification started and will remain in such state (or even get worse!) until all the culprits are identified, repairs for them committed and a new QA verification confirms the branch quality level was restored. The branch can be considered blocked for normal development during all this time.
  
  Is there a CI tool capable of actually **preventing** such regressions from happening, which would perform *pre-commit* QA verifications and allowing commits only when the codebase updated with the respective commits would be passing those pre-commit QA verifications as well, thus guaranteeing a minimum branch quality level?
  
  **Update:** the assumption is that suitable automated QA verifications with appropriate coverage to be able to detect the respective regressions are available for invocation by the CI tool(s).
  
  **Tags:** continuous-integration, software-recommendation
  
  > **Comments:**
  > **username_1**: I keep wondering about the correct way to understand this question (and your own recommendation in your own answer). If I'd replace "monitoring" with "after the facts", and "preventing" with "prvent them from happening", would that more or less be the same question then? Also, maybe you can add some example scenario to explain the difference?  
  > **username_0**: @username_1 Is this any better?  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_0 on 2017-03-03. Score: 0 | Accepted answer
  
  [ApartCI](https://apartci.apartsw.com/) is a CI system designed exactly to prevent regressions, thus guaranteeing flat or increasing branch quality level. Still in beta.
  
  It orchestrates centralized pre-commit verifications in such manner as to ensure that a change is committed only after it is verified, together with all other changes committed before it, to meet or exceed the latest branch quality level.
  
  This is the key difference compared to traditional developer-driven pre-commit verifications, often done in parallel , which leaves room for regressions caused by interfering changes which were never tested together.
  
  The tool is also [designed to easily scale](https://apartci.apartsw.com/help_scale.html) - capable of sustaining very high rates of incoming candidate changes and supporting 100s/1000s of developers working in the same integration branch.
  
  **Disclaimer:** I'm the author of the tool and founder of the company offering it. Apologies for the ad.
  
  > **Comments:**
  > **username_3**: It's good that you added the disclaimer, but personally I consider asking a question and self-answering it by promoting your own company or products to be a form of spam.  
  > **username_3**: I've asked a question on meta whether this is allowed or not: http://meta.devops.stackexchange.com/q/59  
  > **username_2**: SnapCI did this too. RIP.  
  > **username_0**: @username_2, unless I'm missing something SnapCI and its recommended replacement GoCD are both based on post-commit verifications (triggered by polling the SCM), so they're still monitoring-only. Except maybe for PR checks, but unless these checks are completely serialized they can only reduce the regression occurence rates, but not eliminate them completely.  
  > **username_2**: Dan, not polling no, hooks. And to a short-lived PR branch that is not yet merged into trunk/master - https://trunkbaseddevelopment.com/game-changers/#snap-ci-s-per-commit-speculative-mergeability-analysis-2013  
  > **username_0**: Nice idea, but it still is just monitoring: it indicates if the speculative merge would be OK or not. If it's not OK human intervention is needed. And doesn't scale well - it's still done on the local repo, not on the remote one, for each and every commit.  
  
  ---
  
  ### Answer by username_4 on 2017-03-03. Score: 6
  
  As far as I can tell, you're looking for a tool that will **reject** commits that break the build—a CI tool probably won't be able to prevent regressions by actually fixing your code, but it can stop you from adding bad code to the repository.
  
  [Atlassian has a few interesting applications of Git hooks](https://www.atlassian.com/continuous-delivery/git-hooks-continuous-integration):
  
  > Server-side pre-receive hooks are an especially powerful compliment to continuous integration because they can prevent developers from pushing code to master, unless the code meets certain conditions – like elite ninja guardians, protecting it from bad code.
  
  If you're using Git, the hooks are **very** powerful (and there are similar hooks for [SVN](http://svnbook.red-bean.com/nightly/en/svn.reposadmin.create.html), [Mercurial](http://hgbook.red-bean.com/read/handling-repository-events-with-hooks.html) and other version control systems), and you might find it useful to use them to run pre-commit checks.
  
  The Git documentation [has a page on creating a hook to reject pushes if they don't meet a certain criteria](https://git-scm.com/book/en/v2/Customizing-Git-An-Example-Git-Enforced-Policy) which could easily be adapted to this use-case.
  
  However, a lot of people would argue that rejecting commits [is a bad idea](https://softwareengineering.stackexchange.com/questions/294228/automatically-reverting-commits-that-fail-the-build) on a `feature` branch—you'll just be wasting time fighting against your CI system when the build breaks for some reason, instead of actually fixing the bug.
  
  On the `master` branch, it could make sense to reject broken merges, because you might want to ensure it always builds. For a `feature` branch, you **will** inevitably break things, and since the code isn't going into production **now**, it makes more sense just to warn you than actually reject your commit altogether.
  
  > **Comments:**
  > **username_0**: Well, what good is a sw image which builds, but is DOA from testing prospective? Devs can't test their code changes even if they build, so they'd be just as blocked. So in general I'd extend commit rejection to anything that fails a minimum QA check, chosen by balancing 2 conflicting requirements: as high as possible to maximize the number of devs protected from blocking and as low as possible such that the QA verification delays don't slow the process down too much.  
  > **username_5**: An example of this might be the pull request model where you can pose certain restrictions on whether a pull request can be merged or not. For example, we (my company) uses Atlassian Bitbucket Server and there are options to require at least N number of approvals and X number of passing builds for the given branch before a pull request is allowed to be merged. This doesn't outright reject it. But prevents accidental merging when tests fail or other eyes have not seen the code yet.  
  > **username_4**: @username_5: Yeah, I find that quite useful—GitHub also offers [protected branches and checks on pull requests](https://help.github.com/articles/enabling-required-status-checks/), which are often helpful.  
  > **username_6**: One argument for allowing broken code in feature branches is it allows devs to push code they are working on to the repo, even if it's not quite ready yet. This can be useful for sharing code with others for early code/architecture reviews before things are ready to go, help debugging issues, or for someone who is leaving on vacation to put partially done work where others can get to it. For feature branches, I would only put things like linters and as pre-commit hooks.  
  
  ---
  
  ### Answer by username_7 on 2017-03-04. Score: 2
  
  No tool could possibly *guarantee* no regressions - that depends much more on your tests than the tool executing them. However, you can help to prevent regressions that will be caught from entering into the integration branch. You can do this with pre-commit hooks, but it's often easier with pull requests (which hopefully you're already using for peer code review).
  
  If a branch is up-to-date with its upstream (where the PR is merging to), and its tests pass, then they will still pass after the merge; the state of the target branch after the merge will match the state of the source branch before the merge.
  
  It's generally not particularly difficult (depending on the tools used) to indicate both whether the source branch in a PR is up-to-date with the target, and if it has a passing CI build. You can use this as a requirement (by policy, and/or enforced in software) for merging the pull request.
  
  > **Comments:**
  > **username_0**: "If a branch is up-to-date with its upstream (where the PR is merging to), and its tests pass, then they will still pass after the merge" - Why? A merge is a discontinuity, it brings unknowns. Like conflicts - the code might not even build until they're resolved. You **need** to run the tests and confirm that they pass for any branch merge. I'd say even for a fast-forward one, if you want to play it safe. See https://www.apartsw.com/insights/2016/11/16/pre_commit_verification_myths.html  
  > **username_0**: Um, yes, such guarantee is possible, check my answer. Well, maybe I should clarify: by regression I mean results worse that the branch baseline results (and ignoring possibility of false positives, those need to be addressed as they can skew the baseline, derailing the whole thing and requiring human intervention). Intermittent false negatives are just a nuisance, slowing things down, but can be dealt with.  
  > **username_7**: You can't guarantee no regressions, you can only guarantee no detected regressions. If a change causes a regression not caught by a test, that's a regression that a CI tool cannot make any guarantees about. And while merging two sets of changes does bring unknowns, you can choose to do that in the feature branch (by merging the upstream in) before merging the other direction. If the source is up to date with the target, it's a simple merge (fast forward), and afterward the target state will be identical to the source state before the merge, hence if tests passed before, they'll pass after.  
  > **username_0**: Splitting hairs. The CI tool can be configured with a test to detect and thus protect against whichever regression you care about. I won't argue too much about the merges - my goal is to avoid them as much as possible, they're just trouble overall :)  
  > **username_7**: My point is that it's not the CI tool offering that protection, it's you, by writing the tests. The CI tool can't provide any guarantees beyond the tests you provide.  
  > **username_0**: OK, I added a clarification to the question.  
  
  ---
  
  ### Answer by username_8 on 2018-09-17. Score: 1
  
  Use GitLAB, you can set in project-settings to only allow a merge when the pipeline succeed, so can have a truly Continuous Integration, combine that with adding your QA to the list of merge approvals and with Dynamic Environments, you can have quality assurance before you merge to the master.
  
  > **Comments:**
  > **username_0**: That works but only if you do not allow a 2nd merge to start QA checks before the previous merge is completed, otherwise the 2nd merge won't see the 1st one, leaving room for regressions. In other words the merges (including their QA checks) must be completely serialized, which doesn't scale for large teams.  
  
  ---
  
  ### Answer by username_9 on 2017-03-03. Score: 1
  
  True continuous integration tools (as opposed to just continuous testing) like Reitveld and Zuul can help, though they are only as good as the tests you write and code reviews you do.
  
  > **Comments:**
  > **username_0**: But Reitveld appears to be a collaborative code review tool, not a CI tool, am I missing something? This is what I looked at: https://github.com/rietveld-codereview/rietveld  
  > **username_0**: Zuul appears to be indeed, related, I'll study it closer.  
  > **username_9**: It doesn't do testing but it does handle some aspects of branch management, acting as a gatekeeper system, which is the best bet to keep bad code from getting in by blocking the merge.  
  > **username_0**: I see what you mean. Well, it can help with the overall code quality, but by itself doesn't bring any guarantee. Two perfectly good changes that pass all QA verifications independently can still cause breakages when they meet in the branch.  
  
  ---
  
  