Answers: <p>Maybe if you maintained branches per versions instead of per customers it could help reduce their number?</p>
  
  <p>Otherwise the only way to really get away from it is to be able to host the software yourself and switch to a SaaS model where you would be able to maintain only one version of it.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-22T16:05:50.027Z
Answers: Unfortunately our customers often operate in very shut down environments because of the financial data they are working with, so a SaaS model wouldn't be acceptable.
Answers: Mark Wheeler
Answers: CC BY-SA 3.0
Answers: 2018-02-21T15:13:54.180Z
Answers: 2018-02-21T15:13:54.180Z
Answers: Michael Pereira
Answers: <p>As Michael mentioned, offer a standard solution based on release versions/numbers, with a reasonably long lifespan for your industry (maybe interleaved with one or more shorter lifespan intermediate versions, if it makes sense for your typical customers).</p>
  
  <p>Give your customers the option to embark on this standard release track, maybe with a decent migration deadline in place.</p>
  
  <p>If they insist on a full-custom branch support strategy just charge them accordingly, to properly cover all your extra costs for offering such full-custom support - it only makes business sense. Some customers will migrate to reduce their costs (which will help you reduce the number of custom branches), some won't. </p>
  
  <p>Variable support billing, growing progressively with the age of the release versions from which the custom branches originate can also be an incentive for customers to migrate to newer branches faster, helping with faster closing of the older custom branches. This can help reduce the number of custom branches per customer - if you have customers that simultaneouly run multiple versions of your software.</p>
  
  <p>Make sure you don't fall into the trap of doing full branch merges from/to any of the release branches (both standard and custom), all changes to them should be either individually developed or cherry-picked merged fixes. </p>
  
  <p>Because each of these branches will gradually diverge from each-other, the number of hotfixes requiring customisation/individual development will grow exponentially (plain cherry-pick merging will fail). You need to take the development cost for these into account.</p>
  
  <p>With no (significant) branch merges in the picture you can (and should, I cannot stress its importance enough) build fully automated CI/CD pipelines for these branches, accompanied with a good hotfix tracking/management system in place, making the hotfix delivery just routine (or almost).</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-22T16:07:18.450Z
Answers: Dan - so obvious and simple but it makes perfect sense.  Money makes the world go round and in turn should help either compensate us for the cost of long lived branches or encourage customers to upgrade and stay close to trunk. Thanks for you good advice.
Answers: Mark Wheeler
Answers: CC BY-SA 3.0
Answers: 2018-02-21T17:56:37.163Z
Answers: 2018-02-21T17:56:37.163Z
Answers: Dan Cornilescu
Body: <p>How do you avoid a branchageddon situation when working with large organisations?  </p>
  
  <p>We work with a number of large financial organisations whose approach is to not take updates to software, but instead only high/critical security patches and bespoke functionality.  These organisations will only take patches and custom release in between major updates. Major updates can be years apart and carry high costs. This approach causes us (the software house) to have a branch of our code per major customer, which carries all the costs and inefficiencies of long term branching.</p>
  
  <p>My questions to the community are:</p>
  
  <ul>
  <li>Have you experienced similar update acceptance approaches from your
  customers? </li>
  <li>What suggestions do you have to help work with this
  approach? </li>
  <li>What suggestions do you have to help change organisations
  approaches to taking software updates?</li>
  </ul>
  
Comments: CC BY-SA 3.0
Comments: 2018-02-21T14:32:42.683Z
Comments: Hey Mark, it sounds like you have an interesting dilemma. How do you manage developing these per customer updates? Do you develop them on a one-off basis for each customer, or is it something that you develop and apply to all customers?
Comments: Preston Martin
Comments: CC BY-SA 3.0
Comments: 2018-02-21T18:31:34.310Z
Comments: Personally, I might look for other employment in this situation. This sounds like a security incident waiting to happen... I can tell you for the appliance vendor I worked for, they had a major bug that was fixed in an update that they reportedly could not go to. They wanted a custom fix. We refused to build them one and told them that they needed to go fix their business policies - we weren't going to roll a custom hotfix for a bug we had already patched just so they could avoid internal political issues.
Comments: James Shewey
Comments: CC BY-SA 3.0
Comments: 2018-02-22T16:02:09.783Z
Comments: We manage the custom client specific updates via multiple code branches and cross fixing security updates to all branches, and cross fixing branch code back into trunk. Often customer A will not take customer B updates in their branch, they will only take their own updates and security patches.  This is driven by a desire for stability in their branch and so they only have to test updates relevant to them.  They take trunk updates less frequently (i.e. months to years) when they are ready to run complete test reruns, which can take months to complete.  Automated testing could be the answer!
Comments: Mark Wheeler
Tags: <release><process><branch>
Title: How to avoid branchageddon with large organisations?
UserDisplayName: Mark Wheeler
ThreadText: # How to avoid branchageddon with large organisations?
  
  (Asked by: username_0 on 2018-02-21)
  
  How do you avoid a branchageddon situation when working with large organisations?
  
  We work with a number of large financial organisations whose approach is to not take updates to software, but instead only high/critical security patches and bespoke functionality. These organisations will only take patches and custom release in between major updates. Major updates can be years apart and carry high costs. This approach causes us (the software house) to have a branch of our code per major customer, which carries all the costs and inefficiencies of long term branching.
  
  My questions to the community are:
  
  * Have you experienced similar update acceptance approaches from your
    customers?
  * What suggestions do you have to help work with this
    approach?
  * What suggestions do you have to help change organisations
    approaches to taking software updates?
  
  **Tags:** release, process, branch
  
  > **Comments:**
  > **username_1**: Hey Mark, it sounds like you have an interesting dilemma. How do you manage developing these per customer updates? Do you develop them on a one-off basis for each customer, or is it something that you develop and apply to all customers?  
  > **username_2**: Personally, I might look for other employment in this situation. This sounds like a security incident waiting to happen... I can tell you for the appliance vendor I worked for, they had a major bug that was fixed in an update that they reportedly could not go to. They wanted a custom fix. We refused to build them one and told them that they needed to go fix their business policies - we weren't going to roll a custom hotfix for a bug we had already patched just so they could avoid internal political issues.  
  > **username_0**: We manage the custom client specific updates via multiple code branches and cross fixing security updates to all branches, and cross fixing branch code back into trunk. Often customer A will not take customer B updates in their branch, they will only take their own updates and security patches.  This is driven by a desire for stability in their branch and so they only have to test updates relevant to them.  They take trunk updates less frequently (i.e. months to years) when they are ready to run complete test reruns, which can take months to complete.  Automated testing could be the answer!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2018-02-21. Score: 3 | Accepted answer
  
  As username_3 mentioned, offer a standard solution based on release versions/numbers, with a reasonably long lifespan for your industry (maybe interleaved with one or more shorter lifespan intermediate versions, if it makes sense for your typical customers).
  
  Give your customers the option to embark on this standard release track, maybe with a decent migration deadline in place.
  
  If they insist on a full-custom branch support strategy just charge them accordingly, to properly cover all your extra costs for offering such full-custom support - it only makes business sense. Some customers will migrate to reduce their costs (which will help you reduce the number of custom branches), some won't.
  
  Variable support billing, growing progressively with the age of the release versions from which the custom branches originate can also be an incentive for customers to migrate to newer branches faster, helping with faster closing of the older custom branches. This can help reduce the number of custom branches per customer - if you have customers that simultaneouly run multiple versions of your software.
  
  Make sure you don't fall into the trap of doing full branch merges from/to any of the release branches (both standard and custom), all changes to them should be either individually developed or cherry-picked merged fixes.
  
  Because each of these branches will gradually diverge from each-other, the number of hotfixes requiring customisation/individual development will grow exponentially (plain cherry-pick merging will fail). You need to take the development cost for these into account.
  
  With no (significant) branch merges in the picture you can (and should, I cannot stress its importance enough) build fully automated CI/CD pipelines for these branches, accompanied with a good hotfix tracking/management system in place, making the hotfix delivery just routine (or almost).
  
  > **Comments:**
  > **username_0**: Dan - so obvious and simple but it makes perfect sense.  Money makes the world go round and in turn should help either compensate us for the cost of long lived branches or encourage customers to upgrade and stay close to trunk. Thanks for you good advice.  
  
  ---
  
  ### Answer by username_3 on 2018-02-21. Score: 1
  
  Maybe if you maintained branches per versions instead of per customers it could help reduce their number?
  
  Otherwise the only way to really get away from it is to be able to host the software yourself and switch to a SaaS model where you would be able to maintain only one version of it.
  
  > **Comments:**
  > **username_0**: Unfortunately our customers often operate in very shut down environments because of the financial data they are working with, so a SaaS model wouldn't be acceptable.  
  
  ---
  
  