Answers: <p>In the case of off-hours emergency fixes, it's more practical to require less sign-off for changes than your normal procedure. Generally, you can deploy a fix and then do post-approvals the next business day. If the fix is not approved, it can be reverted and replaced with a permanent solution.</p>
  
  <p>During an outage situation, the number one priority should be to restore service. If your organization does not recognize this relaxed process during an outage, then yes, your only option is to start waking more people up for sign off.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-09T08:13:58.377Z
Answers: I agree with your recommendation, which seem to be similar to my own recommendation (answer). Can you think of some example in the SCM world that you are familiar with, and HOW you'd implement it there? If so, can you expand on that in your answer?
Answers: Pierre.Vriens
Answers: CC BY-SA 3.0
Answers: 2017-03-09T02:42:16.150Z
Answers: 2017-03-09T02:42:16.150Z
Answers: tayworm
Answers: <p>In the SCM-world where I'm mostly familiar with, the above scenario is typically addressed by what's called the "<strong><em>abbreviated</em></strong>-approval list procedure.</p>
  
  <p>Here is a blueprint of it:</p>
  
  <ul>
  <li>Define your business hours, say from 8 am to 6 pm.</li>
  <li>Define a <strong><em>complete</em></strong> approval list of (say) 3 levels of approval (for roles X, Y and Z).</li>
  <li>Define an <strong><em>abbreviated</em></strong> approval list of (say) only 1 level of approval (only for roles X).</li>
  <li><strong><em>Planned changes</em></strong> always require all approvals from the complete approval list.</li>
  <li>For <strong><em>Unplanned changes</em></strong>, the complete approval list is used <em>also</em> to gather the required approvals, provided the approvals are to be issued <em>during</em> the defined business hours.</li>
  <li>For any approvals of <strong><em>unplanned changes</em></strong> that are to be issued <em>outside</em> the defined business hours:
  
  <ul>
  <li>Only the approvals from the abbreviated approval list (such as role X above) are required to authorize the change. And after the authorization by the abbreviated approval list is given, the deployment of the change (in the target environment) will actually be performed.</li>
  <li>But additional <strong><em>post</em></strong>-approvals will be needed afterwards (within a reasonable amount of hours/days), i.e from all roles contained in the complete approval list (such as role Y and Z above), which are not also contained in the abbreviated approval list (such as role X above). And if within the (upfront) agreed amount of hours/days not all post-approvals have been issued (e.g because the fix worked "this" time, but was only like a temporary fix), then the change might be subject to a rollback. While there is at least 1 outstanding post-approval, the change is marked as "waiting post approvals".</li>
  </ul></li>
  </ul>
  
  <p>With such solution in place, the call can be closed around <strong>3:23 am</strong> ... since there will be no more red flag at <strong>3:21 am</strong> ... ggggrrreat, time for a beer to celebrate my fix to get production going again (instead of coffee) ... and fingers crossed the outstanding post approvals will come in soon ...</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-08T19:41:57.833Z
Answers: 2017-03-27T09:46:07.793Z
Answers: 2017-03-27T09:46:07.793Z
Answers: Pierre.Vriens
Body: <p>Consider this scenario (any comparison with real world situations is purely by accident):</p>
  
  <ul>
  <li><strong>3:07 am</strong>: incoming support call "<em>Something in production went down, I need your help!</em>".</li>
  <li><strong>3:12 am</strong>: connected to the system (logon accepted) ... and no time for coffee.</li>
  <li><strong>3:15 am</strong>: lucky you, right away you could spot the issue via some error message somewhere.</li>
  <li><strong>3:17 am</strong>: use your SCM toolbox to grab the code, fix the issue, test it, great ... my fix works!</li>
  <li><strong>3:20 am</strong>: get in touch with the <strike>Dev</strike>Ops-team to ship the fix and to get production running again.</li>
  <li><strong>3:21 am</strong>: red flag ... "<em>To respect <a href="/questions/tagged/four-eyes" class="post-tag" title="show questions tagged &#39;four-eyes&#39;" rel="tag">four-eyes</a>, we need 2 more eyes to get approval for this fix</em>".</li>
  <li><strong>3:22 am</strong>: ggggrrrreat, now what, who else can we call (= wake up some manager)?</li>
  </ul>
  
  <p>If you implemented some approval procedure similar to my answer to "<a href="https://devops.stackexchange.com/questions/337/what-are-possible-implementations-or-examples-of-the-four-eyes-principle/338#338">What are possible implementations (or examples) of the four-eyes principle?</a>", then you're out of luck ... here are your choices:</p>
  
  <ul>
  <li>Your fix will be stuck (read: production will be down) until 2 more eyes got involved.</li>
  <li>You figure out a way to get around the missing eyes.</li>
  </ul>
  
  <p><strong><em>So how to implement the four-eyes principle for emergency fixes?</em></strong> ... So that you get production up and running asap, i.e. around  <strong>3:25 am</strong> ... And so that you can also close the call (and go back to where you came from)?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-08T19:58:48.130Z
Comments: You did contact a **team**, which means they should have already blessed the patch with respect to the approbation principles in place. I really start to hate those rhetorical questions :( just my opinion, don't bother with it too much
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-08T20:09:12.820Z
Comments: @Tensibai how can one possibly "have blessed some patch" (or fix) upfront, not knowing what the cause of the problem was when "you were contacted"? Also, can you be more specific about the rhetorical? Not a good fit, something else?
Comments: Pierre.Vriens
Comments: CC BY-SA 3.0
Comments: 2017-03-08T20:18:28.650Z
Comments: I mean you say you were able to contact the team at 3:20, which means it's not just you pushing the fix. I use rhetorical as 'hypothetical case, based from experience or not, where you already know which answer you're waiting for'. More or less my concerns on [meta](http://meta.devops.stackexchange.com/questions/77/dealing-with-too-generic-questions) I feel alone not interested by this 'generic principles' Q/A so I'm probably wrong. What I'm sure off is that I won't visit twice a Q/A about generic principles if I was an outlander of this beta.
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-08T20:20:29.813Z
Comments: I may say the same about the generic Jenkins questions if they were coming now
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-08T21:00:42.457Z
Comments: Commitments won't be computed until public beta, for now in private we're building what we 'committed' users feel are exemplary questions for this site, and I think we have  bunch of work in the remaining 12 days, or I may just have to go through the 7 stages of grief
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-08T21:04:05.080Z
Comments: And the commitment is (IIRC) to write 10 upvoted posts either q or a, I think you've fulfilled it already :) For the share, I'll wait public beta to advertise, each step on its time
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-08T21:13:09.130Z
Comments: Désolé, I'm lost:)  I'm very unsure of what you mean, porting drupal oriented Q here on devops ?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-03-08T21:14:31.200Z
Comments: Maybe we should move to chat, either main or a specific one for this talk, I think main is ok as it's not really related to this question above
Comments: Tensibai
Tags: <configuration><four-eyes>
Title: How to implement the four-eyes principle for emergency fixes?
UserDisplayName: Pierre.Vriens
ThreadText: # How to implement the four-eyes principle for emergency fixes?
  
  (Asked by: username_0 on 2017-03-08)
  
  Consider this scenario (any comparison with real world situations is purely by accident):
  
  * **3:07 am**: incoming support call "*Something in production went down, I need your help!*".
  * **3:12 am**: connected to the system (logon accepted) ... and no time for coffee.
  * **3:15 am**: lucky you, right away you could spot the issue via some error message somewhere.
  * **3:17 am**: use your SCM toolbox to grab the code, fix the issue, test it, great ... my fix works!
  * **3:20 am**: get in touch with the DevOps-team to ship the fix and to get production running again.
  * **3:21 am**: red flag ... "*To respect [four-eyes](/questions/tagged/four-eyes "show questions tagged 'four-eyes'"), we need 2 more eyes to get approval for this fix*".
  * **3:22 am**: ggggrrrreat, now what, who else can we call (= wake up some manager)?
  
  If you implemented some approval procedure similar to my answer to "[What are possible implementations (or examples) of the four-eyes principle?](https://devops.stackexchange.com/questions/337/what-are-possible-implementations-or-examples-of-the-four-eyes-principle/338#338)", then you're out of luck ... here are your choices:
  
  * Your fix will be stuck (read: production will be down) until 2 more eyes got involved.
  * You figure out a way to get around the missing eyes.
  
  ***So how to implement the four-eyes principle for emergency fixes?*** ... So that you get production up and running asap, i.e. around **3:25 am** ... And so that you can also close the call (and go back to where you came from)?
  
  **Tags:** configuration, four-eyes
  
  > **Comments:**
  > **username_1**: You did contact a **team**, which means they should have already blessed the patch with respect to the approbation principles in place. I really start to hate those rhetorical questions :( just my opinion, don't bother with it too much  
  > **username_0**: @username_1 how can one possibly "have blessed some patch" (or fix) upfront, not knowing what the cause of the problem was when "you were contacted"? Also, can you be more specific about the rhetorical? Not a good fit, something else?  
  > **username_1**: I mean you say you were able to contact the team at 3:20, which means it's not just you pushing the fix. I use rhetorical as 'hypothetical case, based from experience or not, where you already know which answer you're waiting for'. More or less my concerns on [meta](http://meta.devops.stackexchange.com/questions/77/dealing-with-too-generic-questions) I feel alone not interested by this 'generic principles' Q/A so I'm probably wrong. What I'm sure off is that I won't visit twice a Q/A about generic principles if I was an outlander of this beta.  
  > **username_1**: I may say the same about the generic Jenkins questions if they were coming now  
  > **username_1**: Commitments won't be computed until public beta, for now in private we're building what we 'committed' users feel are exemplary questions for this site, and I think we have  bunch of work in the remaining 12 days, or I may just have to go through the 7 stages of grief  
  > **username_1**: And the commitment is (IIRC) to write 10 upvoted posts either q or a, I think you've fulfilled it already :) For the share, I'll wait public beta to advertise, each step on its time  
  > **username_1**: Désolé, I'm lost:)  I'm very unsure of what you mean, porting drupal oriented Q here on devops ?  
  > **username_1**: Maybe we should move to chat, either main or a specific one for this talk, I think main is ok as it's not really related to this question above  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2017-03-08. Score: 8 | Accepted answer
  
  In the SCM-world where I'm mostly familiar with, the above scenario is typically addressed by what's called the "***abbreviated***-approval list procedure.
  
  Here is a blueprint of it:
  
  * Define your business hours, say from 8 am to 6 pm.
  * Define a ***complete*** approval list of (say) 3 levels of approval (for roles X, Y and Z).
  * Define an ***abbreviated*** approval list of (say) only 1 level of approval (only for roles X).
  * ***Planned changes*** always require all approvals from the complete approval list.
  * For ***Unplanned changes***, the complete approval list is used *also* to gather the required approvals, provided the approvals are to be issued *during* the defined business hours.
  * For any approvals of ***unplanned changes*** that are to be issued *outside* the defined business hours:
    + Only the approvals from the abbreviated approval list (such as role X above) are required to authorize the change. And after the authorization by the abbreviated approval list is given, the deployment of the change (in the target environment) will actually be performed.
    + But additional ***post***-approvals will be needed afterwards (within a reasonable amount of hours/days), i.e from all roles contained in the complete approval list (such as role Y and Z above), which are not also contained in the abbreviated approval list (such as role X above). And if within the (upfront) agreed amount of hours/days not all post-approvals have been issued (e.g because the fix worked "this" time, but was only like a temporary fix), then the change might be subject to a rollback. While there is at least 1 outstanding post-approval, the change is marked as "waiting post approvals".
  
  With such solution in place, the call can be closed around **3:23 am** ... since there will be no more red flag at **3:21 am** ... ggggrrreat, time for a beer to celebrate my fix to get production going again (instead of coffee) ... and fingers crossed the outstanding post approvals will come in soon ...
  
  
  ---
  
  ### Answer by username_2 on 2017-03-09. Score: 3
  
  In the case of off-hours emergency fixes, it's more practical to require less sign-off for changes than your normal procedure. Generally, you can deploy a fix and then do post-approvals the next business day. If the fix is not approved, it can be reverted and replaced with a permanent solution.
  
  During an outage situation, the number one priority should be to restore service. If your organization does not recognize this relaxed process during an outage, then yes, your only option is to start waking more people up for sign off.
  
  > **Comments:**
  > **username_0**: I agree with your recommendation, which seem to be similar to my own recommendation (answer). Can you think of some example in the SCM world that you are familiar with, and HOW you'd implement it there? If so, can you expand on that in your answer?  
  
  ---
  
  