Answers: <p>You've highlighted the great conundrum of feature flags!</p>
  <p>I don't believe it would be possible to check this with static-analysis alone as it requires some knowledge of application behaviour (even if the understanding is &quot;what is a feature flag&quot;, as they often look like things that aren't feature flags).</p>
  <p>In order to prove everything is hidden by feature flags, you'd need to be able to run a test suite aginst the application with all flags switched off.</p>
  <p>Whether you also need to test &quot;on&quot; states and combinations of &quot;on&quot; states depends a lot of how you are using feature flags. When you use them to hide a feature in development it's very different to when you make a system user-configurable with flags, or have multiple values per flag.</p>
  <p>More flags and flag values = way more combinations. It quickly becomes possible to only test &quot;selected combinations&quot;.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-03-16T09:23:39.833Z
Answers: 2023-03-16T09:23:39.833Z
Answers: Fenton
Body: <p>In a CD environment with new features hidden behind feature flags, there must be a way to ensure that code being pushed to production is effectively zero change?</p>
  <p>Particularly in a change management controlled environment, it would be ideal to not treat code deployment as a change - only the feature switch on. To be able to do this, is there a way to prove that all code changes are protected behind feature flags? And that the effective flow is zero change. Feels like something that could and should exist. But maybe others solve this differently?</p>
  
Comments: CC BY-SA 4.0
Comments: 2023-03-13T12:11:34.607Z
Comments: Perhaps I'm misunderstanding, but this seems like the reason why you would use automated testing for regression. If you have sufficient test coverage in your automated regression suite and the only changes are behind a feature flag, all of the automated tests would pass. They would fail if there was an unexpected change outside of the feature flag.
Comments: Thomas Owens
Comments: CC BY-SA 4.0
Comments: 2023-03-13T23:11:20.873Z
Comments: Well, yes, maybe. That was my initial thinking, but it kind of raised more questions than answers for me. In a microservices setup, where you want to keep each microservice independently deployable, there's 3 levels you could test this. Unit tests - but they're being changed with the code, so too likely to break at the same time. Component/service/contract tests - they always seem the least reliable, as they're in the no-mans land between tech/dev and qa/product. Or Acceptance tests on the whole system - too slow and brittle. Hence why I was hoping SA could help
Comments: james-tru
Comments: CC BY-SA 4.0
Comments: 2023-03-13T23:36:31.577Z
Comments: I want to think this through a little more. Are you concerned with the case where a feature flag affects more than one service's behavior? That seems to be the problematic case. Although I also think you may be missing one level of tests - you have some tests that test a service in isolation and other tests that test the integration between two or more services (but not yet at an acceptance level). I guess I'm struggling with why static analysis would be helpful, especially since I don't see how the tool would look across service boundries to find the types of issues you're concerned with.
Comments: Thomas Owens
Comments: CC BY-SA 4.0
Comments: 2023-03-14T22:34:02.093Z
Comments: So, say we have a simple microservice that had a new feature added: `if(handle_passover_flag) { do_something_slightly_different_on_passover_if_its_awednesday() } else { do_normal_thing_except_on_wednesdays()}` where the new feature is a pretty subtle edge case. It's the kind of thing component tests often miss. The person writing the new feature could accidentally change some existing code like  `common.utilities.is_weird_day()`. Static analysis could compile with the flag as false and then see that the compiled old code and new code were the same and only promote if it is
Comments: james-tru
Comments: CC BY-SA 4.0
Comments: 2023-03-15T12:41:29.463Z
Comments: I don't think this is feasible. First, you're limited to compiled languages. And then you need to take the time to do multiple compilations, which would increase in number as you increase the number of feature flags, which adds time to your build pipeline. You wouldn't need complex analysis at this point - just a diff. But I also don't think you need component tests - if there was a change to `common.utilities.is_weird_day()`, then your unit tests should fail if an expected case is no longer properly managed.
Comments: Thomas Owens
Tags: <continuous-delivery><feature-flags><feature-toggles>
Title: Static Analysis for Feature Flag Safety
UserDisplayName: james-tru
ThreadText: # Static Analysis for Feature Flag Safety
  
  (Asked by: username_0 on 2023-03-13)
  
  In a CD environment with new features hidden behind feature flags, there must be a way to ensure that code being pushed to production is effectively zero change?
  
  Particularly in a change management controlled environment, it would be ideal to not treat code deployment as a change - only the feature switch on. To be able to do this, is there a way to prove that all code changes are protected behind feature flags? And that the effective flow is zero change. Feels like something that could and should exist. But maybe others solve this differently?
  
  **Tags:** continuous-delivery, feature-flags, feature-toggles
  
  > **Comments:**
  > **username_1**: Perhaps I'm misunderstanding, but this seems like the reason why you would use automated testing for regression. If you have sufficient test coverage in your automated regression suite and the only changes are behind a feature flag, all of the automated tests would pass. They would fail if there was an unexpected change outside of the feature flag.  
  > **username_0**: Well, yes, maybe. That was my initial thinking, but it kind of raised more questions than answers for me. In a microservices setup, where you want to keep each microservice independently deployable, there's 3 levels you could test this. Unit tests - but they're being changed with the code, so too likely to break at the same time. Component/service/contract tests - they always seem the least reliable, as they're in the no-mans land between tech/dev and qa/product. Or Acceptance tests on the whole system - too slow and brittle. Hence why I was hoping SA could help  
  > **username_1**: I want to think this through a little more. Are you concerned with the case where a feature flag affects more than one service's behavior? That seems to be the problematic case. Although I also think you may be missing one level of tests - you have some tests that test a service in isolation and other tests that test the integration between two or more services (but not yet at an acceptance level). I guess I'm struggling with why static analysis would be helpful, especially since I don't see how the tool would look across service boundries to find the types of issues you're concerned with.  
  > **username_0**: So, say we have a simple microservice that had a new feature added: `if(handle_passover_flag) { do_something_slightly_different_on_passover_if_its_awednesday() } else { do_normal_thing_except_on_wednesdays()}` where the new feature is a pretty subtle edge case. It's the kind of thing component tests often miss. The person writing the new feature could accidentally change some existing code like  `common.utilities.is_weird_day()`. Static analysis could compile with the flag as false and then see that the compiled old code and new code were the same and only promote if it is  
  > **username_1**: I don't think this is feasible. First, you're limited to compiled languages. And then you need to take the time to do multiple compilations, which would increase in number as you increase the number of feature flags, which adds time to your build pipeline. You wouldn't need complex analysis at this point - just a diff. But I also don't think you need component tests - if there was a change to `common.utilities.is_weird_day()`, then your unit tests should fail if an expected case is no longer properly managed.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-03-16. Score: 0
  
  You've highlighted the great conundrum of feature flags!
  
  I don't believe it would be possible to check this with static-analysis alone as it requires some knowledge of application behaviour (even if the understanding is "what is a feature flag", as they often look like things that aren't feature flags).
  
  In order to prove everything is hidden by feature flags, you'd need to be able to run a test suite aginst the application with all flags switched off.
  
  Whether you also need to test "on" states and combinations of "on" states depends a lot of how you are using feature flags. When you use them to hide a feature in development it's very different to when you make a system user-configurable with flags, or have multiple values per flag.
  
  More flags and flag values = way more combinations. It quickly becomes possible to only test "selected combinations".
  
  
  ---
  
  