Answers: <p>If you need to remove features from something that has been deployed to <code>preprod</code>, the new code should go through the process of promotion from <code>dev</code> to <code>qa</code> to <code>preprod</code> before it reaches <code>prod</code>. Depending on your amount and quality of automated testing, this could be a very fast process, but it doesn't make sense to short-circuit the process and put potentially untested or undertested changes into production.</p>
  <p>In the absence of something like feature flags or keystone interfaces, <a href="https://devops.stackexchange.com/a/18605/441">Levi's suggestion of feature branches and squashing merges to make it easier to revert all aspects of a change</a> is good. However, it doesn't negate the need for good testing since later changes could rely on the earlier (now-reverted) change. However, I would also invest time in understanding the root causes for needing to revert the change in the first place and addressing those so this becomes an exceptional case.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-11-28T13:07:32.290Z
Answers: When you say *invest time to understand the root causes...* do you have things like "too little time to test appropriately" or "underestimates time for development" on mind? or something completely different? PS. feature flags & keystone interfaces are very nice hint here!
Answers: tlt
Answers: CC BY-SA 4.0
Answers: 2023-11-28T14:05:33.317Z
Answers: @dzg I don't know what your root causes could be. There are techniques, such as using five whys or Ishikawa diagrams or causal graphs to understand why you had to revert the changes. Creating a value stream map could also be beneficial. Unfortunately, without a much deeper understanding of your process, I couldn't assess potential problems or causes for needing to revert changes late in the development process.
Answers: Thomas Owens
Answers: CC BY-SA 4.0
Answers: 2023-11-29T18:07:39.810Z
Answers: Yeah, I see. That make sense. Btw, does the same line of thinking like in your answer stand for the case when there happen to be 2 competing releases? For example, it took longed than expected to test one release in *preprod* while other is ready to go to *qa*. Does in that case 1st release branch gets invalidated and new one created with both change sets going again through *qa > preprod > prod* cycle?
Answers: tlt
Answers: CC BY-SA 4.0
Answers: 2023-11-29T18:31:31.900Z
Answers: @dzg That depends on why you have two competing releases. If you have different production environments, then you should have parallel qa and preprod environments as well. If you have a single production environment, If you have a single production environment, I would want to focus on reducing the time to test in each environment and getting to a series of releases. I'd really need to dig into why you're finding an issue in preprod if it got through dev and qa activities, as well.
Answers: Thomas Owens
Answers: CC BY-SA 4.0
Answers: 2023-11-29T18:37:21.690Z
Answers: yeah, its web app and its a single production environment. I would say all of the problems stem from less-than-ideal planning. For example: release dates are set strict without taking into account ETAs from developers enough, time planned for testing was too short so it turned out that QA didn't pick all of the problems and then preprod feedback>hotfix cycle took more than expected. Its that kind of stuff. And sure, that's what needs to be fixed but what I am looking into is what are the technical ways not to solve but to absorb at least some of those issues. If that makes sense.
Answers: tlt
Answers: CC BY-SA 4.0
Answers: 2023-11-29T18:50:47.783Z
Answers: @dzg These are not technical problems, so you won't find technical solutions to them. For example, setting strict release dates. Assuming a fixed team, you need to make either scope or quality flexible. Technical practices may help slightly, but won't fix root causes.
Answers: Thomas Owens
Answers: CC BY-SA 4.0
Answers: 2023-11-29T19:10:28.250Z
Answers: yes, i am completely aware of that and all of this is trying to find ways to "make things slightly better and help slightly".
Answers: tlt
Answers: CC BY-SA 4.0
Answers: 2023-11-28T12:41:20.547Z
Answers: 2023-11-28T12:41:20.547Z
Answers: Thomas Owens
Answers: <p>Have your developers do their work in feature branches and do a single feature per branch. Then merge their work into your <code>dev</code> branch. Now when you need to drop a feature from your release branch it's easier. If you want to make it really easy, when you merge from a feature branch to <code>dev</code> use the squash setting.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-11-28T12:01:37.320Z
Answers: 2023-11-28T12:01:37.320Z
Answers: Levi
Body: <p>We use gitflow branching strategy for different reasons, one of them being complicated bureaucracy around releases.</p>
  <p>We have environments: <code>dev</code>, <code>qa</code>, <code>preprod</code>, <code>prod</code>.</p>
  <p>Once the developers finish their work in <code>dev</code> branch, we create <code>release</code> branch and trigger deployment to <code>qa</code>.
  After <code>qa</code> process is done, we trigger deployment of the same branch to <code>preprod</code>.
  After tests in <code>preprod</code> are finished, we deploy the same branch to <code>prod</code>.</p>
  <p>Issues arise when business asks to remove/add some features from <code>release</code> branch, for example, after deploying to <code>preprod</code> but before deploying to <code>prod</code>. In that case, if its a  simple commit or several of them, we can cherry-pick. But that gets complicated very soon.</p>
  <p>Is there a general solution for this? And, is the solution technical or is it on the planning side like establishing a consensus: <code>release</code> branch doesn't get deployed to <code>production</code> until all the features that went into <code>release</code> branch initially are ready?</p>
  
Tags: <git>
Title: How to organize Gitflow releases with multiple environments for web application?
UserDisplayName: tlt
ThreadText: # How to organize Gitflow releases with multiple environments for web application?
  
  (Asked by: username_0 on 2023-11-28)
  
  We use gitflow branching strategy for different reasons, one of them being complicated bureaucracy around releases.
  
  We have environments: `dev`, `qa`, `preprod`, `prod`.
  
  Once the developers finish their work in `dev` branch, we create `release` branch and trigger deployment to `qa`.
  After `qa` process is done, we trigger deployment of the same branch to `preprod`.
  After tests in `preprod` are finished, we deploy the same branch to `prod`.
  
  Issues arise when business asks to remove/add some features from `release` branch, for example, after deploying to `preprod` but before deploying to `prod`. In that case, if its a simple commit or several of them, we can cherry-pick. But that gets complicated very soon.
  
  Is there a general solution for this? And, is the solution technical or is it on the planning side like establishing a consensus: `release` branch doesn't get deployed to `production` until all the features that went into `release` branch initially are ready?
  
  **Tags:** git
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-11-28. Score: 1 | Accepted answer
  
  If you need to remove features from something that has been deployed to `preprod`, the new code should go through the process of promotion from `dev` to `qa` to `preprod` before it reaches `prod`. Depending on your amount and quality of automated testing, this could be a very fast process, but it doesn't make sense to short-circuit the process and put potentially untested or undertested changes into production.
  
  In the absence of something like feature flags or keystone interfaces, [username_2's suggestion of feature branches and squashing merges to make it easier to revert all aspects of a change](https://devops.stackexchange.com/a/18605/441) is good. However, it doesn't negate the need for good testing since later changes could rely on the earlier (now-reverted) change. However, I would also invest time in understanding the root causes for needing to revert the change in the first place and addressing those so this becomes an exceptional case.
  
  > **Comments:**
  > **username_0**: When you say *invest time to understand the root causes...* do you have things like "too little time to test appropriately" or "underestimates time for development" on mind? or something completely different? PS. feature flags & keystone interfaces are very nice hint here!  
  > **username_1**: @dzg I don't know what your root causes could be. There are techniques, such as using five whys or Ishikawa diagrams or causal graphs to understand why you had to revert the changes. Creating a value stream map could also be beneficial. Unfortunately, without a much deeper understanding of your process, I couldn't assess potential problems or causes for needing to revert changes late in the development process.  
  > **username_0**: Yeah, I see. That make sense. Btw, does the same line of thinking like in your answer stand for the case when there happen to be 2 competing releases? For example, it took longed than expected to test one release in *preprod* while other is ready to go to *qa*. Does in that case 1st release branch gets invalidated and new one created with both change sets going again through *qa > preprod > prod* cycle?  
  > **username_1**: @dzg That depends on why you have two competing releases. If you have different production environments, then you should have parallel qa and preprod environments as well. If you have a single production environment, If you have a single production environment, I would want to focus on reducing the time to test in each environment and getting to a series of releases. I'd really need to dig into why you're finding an issue in preprod if it got through dev and qa activities, as well.  
  > **username_0**: yeah, its web app and its a single production environment. I would say all of the problems stem from less-than-ideal planning. For example: release dates are set strict without taking into account ETAs from developers enough, time planned for testing was too short so it turned out that QA didn't pick all of the problems and then preprod feedback>hotfix cycle took more than expected. Its that kind of stuff. And sure, that's what needs to be fixed but what I am looking into is what are the technical ways not to solve but to absorb at least some of those issues. If that makes sense.  
  > **username_1**: @dzg These are not technical problems, so you won't find technical solutions to them. For example, setting strict release dates. Assuming a fixed team, you need to make either scope or quality flexible. Technical practices may help slightly, but won't fix root causes.  
  > **username_0**: yes, i am completely aware of that and all of this is trying to find ways to "make things slightly better and help slightly".  
  
  ---
  
  ### Answer by username_2 on 2023-11-28. Score: 1
  
  Have your developers do their work in feature branches and do a single feature per branch. Then merge their work into your `dev` branch. Now when you need to drop a feature from your release branch it's easier. If you want to make it really easy, when you merge from a feature branch to `dev` use the squash setting.
  
  
  ---
  
  