Answers: <p>There is definite value in automating manual tasks and placing systems under configuration management that is done using code, not using paper and human intervention.</p>
  
  <p>One huge benefit is the reduction in the amount of rework. You can consider any problem caused by a human mistake in repeating the same sequence a case of rework. As well as any bug that needs to be fixed more than once.</p>
  
  <p>Rework is costly, and there has been some research done to quantify its cost. You can read about it in this whitepaper - <a href="https://devops-research.com/roi/" rel="nofollow noreferrer">https://devops-research.com/roi/</a></p>
  
  <p>The same whitepaper also quantifies the cost of failure for services/systems, and the slowness to resolve failure. All of which are greatly reduced by having a robust automated process that prevents human mistakes from ending up in production systems.</p>
  
  <p>The topic of automating manual tasks has been also investigated in the <em>State of DevOps Report</em> that has been issued every year since 2013. Most of these are available at <a href="https://devops-research.com/research.html" rel="nofollow noreferrer">https://devops-research.com/research.html</a> as well (with the exception of 2013). In every report it was mentioned as the biggest driver for excellence in high-performing organizations of all sizes. There are plenty of reasons and explanations that can be used as proof for your case.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-06-18T08:46:18.403Z
Answers: 2017-06-23T10:34:20.420Z
Answers: 2017-06-23T10:34:20.420Z
Answers: Evgeny Zislis
Answers: <p>The main advantage to automation isn't just automation itself, but also that it provides you with an identical configuration across all of your resources.</p>
  
  <p>Say, for example, you stand up a webserver hosting your application and all relevant configs.  Then you register it as an Amazon AMI so you can deploy multiple copies (say, one for dev, one for test, and one for production).  You roll it out, and with how AWS works, you don't even need to change any IPs by hand.</p>
  
  <p>A few weeks later you realize you need to change SSL session cache size in nginx, or users of your application can't upload large files.</p>
  
  <p>Since you only have a few boxes, you do the update by hand.  You do these config changes a few more times and well-document them.</p>
  
  <p>Then a week later you deploy a new box (a customer is asking for a UAT environment to test their integration).  You're sick, so someone else is standing up the server instead of you.  Or maybe your instance in AWS dies, so you have to rebuild production in the middle of the night after a good binge-drinking session.  Everything works perfectly... except for one specific config-related feature.  Cue wasting two hours of figuring out why it's happening as an account manager is riding your ass, only to realize the other admin forgot a single stupid config setting when deploying a new server.</p>
  
  <p>Have this happen to you a few times, and you'll want to find a way to minimize these issues.</p>
  
  <p>So why not take a few days of your time and write out a simple deployment pipeline with Ansible or Puppet and put it in git?  From that point on, any config changes are only a push away, and you'll rarely have to worry about a similar incident occurring.</p>
  
  <p>Now, it's not peer-reviewed, APA-cited research papers you're asking for, but point me a sysadmin that's never had this happen to them, and my company will probably throw them a job offer with a pretty nice signing bonus.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-06-18T06:39:26.507Z
Answers: *Now, it's not peer-reviewed, APA-cited research papers you're asking for, but point me a sysadmin that's never had this happen to them* - Unfortunately, that's not always good enough proof for upper management.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-18T17:31:29.443Z
Answers: Curious comment, @JamesShewey.  My experience is that upper management isn't who needs to be persuaded that automation has advantaged.  If anything, they have to be persuaded that it's (initially, at least) more complex and nuanced than they imagine it to be.
Answers: Michael - sqlbot
Answers: CC BY-SA 3.0
Answers: 2017-06-19T07:22:05.633Z
Answers: I heared the _"but upper management will not agree"_ argument multiple times for different reasons. In most cases it is imaginary, meaning that people invent the argument without any actual manager saying anything of the like. It is an excuse to not do the proper thing because of fear that someone else might not approve, without actually asking the other person for their approval and/or explaining why it is important to do things properly.
Answers: Evgeny Zislis
Answers: CC BY-SA 3.0
Answers: 2017-06-19T14:24:46.307Z
Answers: The point here (and of this question) is that executive/manager buy-in can nearly always be achieved if you can make your business case. The best way to do so is to prove that something is better for the business with unequivocal,  cold, hard facts. Company cultures vary widely. Some trust their experts, others don't. Some trust their experts, but lack agility and are poor at change. And by "upper" I really just mean "above me" perhaps your manager isn't willing to stick his neck out for your hair brained idea. Point is real data will always win.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-19T22:01:19.237Z
Answers: In most organizations, managers are not brain-dead computers and tend to evaluate ideas presented to them for their merit. Often they do it even better than the person presenting the idea. When this evaluation clearly displays value, in the eyes of the manager, he will usually not resist it too much. Relevant video on this matter - https://youtu.be/hcz1aZ60k7w
Answers: Evgeny Zislis
Answers: CC BY-SA 3.0
Answers: 2017-06-18T06:21:17.950Z
Answers: 2017-06-18T06:21:17.950Z
Answers: maplebird
Answers: <p>One of the things I have the hardest time getting across to people is that the man-hours saved doing repetitive tasks is often only a small part of the value of automation. The bigger part is often hard to measure, and nearly impossible to estimate when automating something for the first time: <em>how it changes the way you work</em>. Talking with developers, this is a lot easier, because they know test automation (unit tests) and it's an easy comparison to make. It's things like:</p>
  
  <ul>
  <li>How does your workflow change when you can do X as often as you want with negligible costs and in very little time?
  
  <ul>
  <li>For unit tests, it means bugs get caught much, much earlier - you can test <em>everything</em> multiple times per day, instead of shipping something to QA once a week. Turnaround on iterations gets orders of magnitude shorter.</li>
  <li>For infrastructure, it means that doing a short-lived test environment or proof of concept is a non-event, as is scaling out, as is replacing an unhealthy instance. The LoE nears zero.</li>
  </ul></li>
  <li>How does your workflow change when you think about automation as part of your process?
  
  <ul>
  <li>For unit tests, this means you design your code differently, to facilitate automation, which (often) leads to better, cleaner code.</li>
  <li>For infrastructure, this encourages you to develop a standard plan for how you're going to automate things - when a new service is being developed, you know what questions you need to ask about it.</li>
  <li>Developers start to change the way they develop as well, thinking about keeping stateless, graceful shutdown, fast restarts, how configuration is handled, keeping log or data file paths configurable, thinking about what paths the application user needs access to and what access it needs, etc.</li>
  </ul></li>
  </ul>
  
  <p>The "pets vs. cattle" shift was driven in large part by automation; when creating a working server is a non-event, then servers become fully fungible and you no longer need to care about individual instances.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-06-19T19:08:40.027Z
Answers: *For unit tests, it means bugs get caught much, much earlier - you can test everything multiple times per day, instead of shipping something to QA once a week.* and *For infrastructure, it means that replacing an unhealthy instance is a non-event* - surely these have measurable, study-able business impacts about which data can be provided in qualitative or quantitative studies.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-19T19:10:20.130Z
Answers: Yes, but the impacts also vary greatly over time, between teams, between projects, and between companies, making the result of any measurement or study outside your own specific situation of limited value.
Answers: Adrian
Answers: CC BY-SA 3.0
Answers: 2017-06-19T21:10:07.550Z
Answers: I thought they called that "standard deviation" in stats terms...
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-19T21:31:42.010Z
Answers: I'm not sure I see your point, or how it relates to the answer.
Answers: Adrian
Answers: CC BY-SA 3.0
Answers: 2017-06-19T22:47:25.183Z
Answers: My point is that qualitative studies understand that your mileage may vary between companies, teams and project and study results represent an *average* result. Studies which have inconclusive data either because results vary too greatly simply have a null finding and conclude that you can't conclude anything for sure or that more study is required. This finding in and of itself is valuable because it would imply that DevOps may not actually be beneficial and it *isn't* a sure bet for improvement.
Answers: James Shewey
Answers: CC BY-SA 3.0
Answers: 2017-06-19T18:37:45.587Z
Answers: 2017-06-19T18:37:45.587Z
Answers: Adrian
Body: <p>Obviously, Automated configuration management and deployment makes sense at large scales when you need to manage hundreds of servers and you need those servers to be identically configured to a consistent standard.</p>
  
  <p>However, in situations where systems are small and need only 2, 3 or 4 servers, it can often be faster in build time to simply stand up a first instance and clone subsequent instances - changing IP addresses and hostnames by hand.</p>
  
  <p>The theoretical advantage to automated deployment and configuration management is that code can be deployed faster, tested faster and services are provided more reliably by automating out the problem of human error. Future upgrades of the underlying OS (eg, CentOS 6 -> CentOS 7) are better/faster and in the long run the above benefits outweigh the initial larger investment of programmatically automating deployments using a configuration management system?</p>
  
  <p>Are there any real-world non-anecdotal studies and data-points to back each of the above claimed benefits to small shops with very low numbers of servers to manage?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-06-18T19:33:25.957Z
Comments: Companies that do formal research on this type of question are far beyond the point that we'd all agree automation is beneficial for them. You'd need to find a university student who is interested in the subject and is able to find small companies willing to eat the costs of participating in this sort of research, which seems unlikely.
Comments: Xiong Chiamiov
Tags: <automation><configuration-management>
Title: Does full automation still provide a better ROI at small scales?
UserDisplayName: James Shewey
ThreadText: # Does full automation still provide a better ROI at small scales?
  
  (Asked by: username_0 on 2017-06-18)
  
  Obviously, Automated configuration management and deployment makes sense at large scales when you need to manage hundreds of servers and you need those servers to be identically configured to a consistent standard.
  
  However, in situations where systems are small and need only 2, 3 or 4 servers, it can often be faster in build time to simply stand up a first instance and clone subsequent instances - changing IP addresses and hostnames by hand.
  
  The theoretical advantage to automated deployment and configuration management is that code can be deployed faster, tested faster and services are provided more reliably by automating out the problem of human error. Future upgrades of the underlying OS (eg, CentOS 6 -> CentOS 7) are better/faster and in the long run the above benefits outweigh the initial larger investment of programmatically automating deployments using a configuration management system?
  
  Are there any real-world non-anecdotal studies and data-points to back each of the above claimed benefits to small shops with very low numbers of servers to manage?
  
  **Tags:** automation, configuration-management
  
  > **Comments:**
  > **username_1**: Companies that do formal research on this type of question are far beyond the point that we'd all agree automation is beneficial for them. You'd need to find a university student who is interested in the subject and is able to find small companies willing to eat the costs of participating in this sort of research, which seems unlikely.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2017-06-18. Score: 8
  
  There is definite value in automating manual tasks and placing systems under configuration management that is done using code, not using paper and human intervention.
  
  One huge benefit is the reduction in the amount of rework. You can consider any problem caused by a human mistake in repeating the same sequence a case of rework. As well as any bug that needs to be fixed more than once.
  
  Rework is costly, and there has been some research done to quantify its cost. You can read about it in this whitepaper - <https://devops-research.com/roi/>
  
  The same whitepaper also quantifies the cost of failure for services/systems, and the slowness to resolve failure. All of which are greatly reduced by having a robust automated process that prevents human mistakes from ending up in production systems.
  
  The topic of automating manual tasks has been also investigated in the *State of DevOps Report* that has been issued every year since 2013. Most of these are available at <https://devops-research.com/research.html> as well (with the exception of 2013). In every report it was mentioned as the biggest driver for excellence in high-performing organizations of all sizes. There are plenty of reasons and explanations that can be used as proof for your case.
  
  
  ---
  
  ### Answer by username_3 on 2017-06-18. Score: 4
  
  The main advantage to automation isn't just automation itself, but also that it provides you with an identical configuration across all of your resources.
  
  Say, for example, you stand up a webserver hosting your application and all relevant configs. Then you register it as an Amazon AMI so you can deploy multiple copies (say, one for dev, one for test, and one for production). You roll it out, and with how AWS works, you don't even need to change any IPs by hand.
  
  A few weeks later you realize you need to change SSL session cache size in nginx, or users of your application can't upload large files.
  
  Since you only have a few boxes, you do the update by hand. You do these config changes a few more times and well-document them.
  
  Then a week later you deploy a new box (a customer is asking for a UAT environment to test their integration). You're sick, so someone else is standing up the server instead of you. Or maybe your instance in AWS dies, so you have to rebuild production in the middle of the night after a good binge-drinking session. Everything works perfectly... except for one specific config-related feature. Cue wasting two hours of figuring out why it's happening as an account manager is riding your ass, only to realize the other admin forgot a single stupid config setting when deploying a new server.
  
  Have this happen to you a few times, and you'll want to find a way to minimize these issues.
  
  So why not take a few days of your time and write out a simple deployment pipeline with Ansible or Puppet and put it in git? From that point on, any config changes are only a push away, and you'll rarely have to worry about a similar incident occurring.
  
  Now, it's not peer-reviewed, APA-cited research papers you're asking for, but point me a sysadmin that's never had this happen to them, and my company will probably throw them a job offer with a pretty nice signing bonus.
  
  > **Comments:**
  > **username_0**: *Now, it's not peer-reviewed, APA-cited research papers you're asking for, but point me a sysadmin that's never had this happen to them* - Unfortunately, that's not always good enough proof for upper management.  
  > **username_4**: Curious comment, @username_0.  My experience is that upper management isn't who needs to be persuaded that automation has advantaged.  If anything, they have to be persuaded that it's (initially, at least) more complex and nuanced than they imagine it to be.  
  > **username_2**: I heared the _"but upper management will not agree"_ argument multiple times for different reasons. In most cases it is imaginary, meaning that people invent the argument without any actual manager saying anything of the like. It is an excuse to not do the proper thing because of fear that someone else might not approve, without actually asking the other person for their approval and/or explaining why it is important to do things properly.  
  > **username_0**: The point here (and of this question) is that executive/manager buy-in can nearly always be achieved if you can make your business case. The best way to do so is to prove that something is better for the business with unequivocal,  cold, hard facts. Company cultures vary widely. Some trust their experts, others don't. Some trust their experts, but lack agility and are poor at change. And by "upper" I really just mean "above me" perhaps your manager isn't willing to stick his neck out for your hair brained idea. Point is real data will always win.  
  > **username_2**: In most organizations, managers are not brain-dead computers and tend to evaluate ideas presented to them for their merit. Often they do it even better than the person presenting the idea. When this evaluation clearly displays value, in the eyes of the manager, he will usually not resist it too much. Relevant video on this matter - https://youtu.be/hcz1aZ60k7w  
  
  ---
  
  ### Answer by username_5 on 2017-06-19. Score: 4
  
  One of the things I have the hardest time getting across to people is that the man-hours saved doing repetitive tasks is often only a small part of the value of automation. The bigger part is often hard to measure, and nearly impossible to estimate when automating something for the first time: *how it changes the way you work*. Talking with developers, this is a lot easier, because they know test automation (unit tests) and it's an easy comparison to make. It's things like:
  
  * How does your workflow change when you can do X as often as you want with negligible costs and in very little time?
    + For unit tests, it means bugs get caught much, much earlier - you can test *everything* multiple times per day, instead of shipping something to QA once a week. Turnaround on iterations gets orders of magnitude shorter.
    + For infrastructure, it means that doing a short-lived test environment or proof of concept is a non-event, as is scaling out, as is replacing an unhealthy instance. The LoE nears zero.
  * How does your workflow change when you think about automation as part of your process?
    + For unit tests, this means you design your code differently, to facilitate automation, which (often) leads to better, cleaner code.
    + For infrastructure, this encourages you to develop a standard plan for how you're going to automate things - when a new service is being developed, you know what questions you need to ask about it.
    + Developers start to change the way they develop as well, thinking about keeping stateless, graceful shutdown, fast restarts, how configuration is handled, keeping log or data file paths configurable, thinking about what paths the application user needs access to and what access it needs, etc.
  
  The "pets vs. cattle" shift was driven in large part by automation; when creating a working server is a non-event, then servers become fully fungible and you no longer need to care about individual instances.
  
  > **Comments:**
  > **username_0**: *For unit tests, it means bugs get caught much, much earlier - you can test everything multiple times per day, instead of shipping something to QA once a week.* and *For infrastructure, it means that replacing an unhealthy instance is a non-event* - surely these have measurable, study-able business impacts about which data can be provided in qualitative or quantitative studies.  
  > **username_5**: Yes, but the impacts also vary greatly over time, between teams, between projects, and between companies, making the result of any measurement or study outside your own specific situation of limited value.  
  > **username_0**: I thought they called that "standard deviation" in stats terms...  
  > **username_5**: I'm not sure I see your point, or how it relates to the answer.  
  > **username_0**: My point is that qualitative studies understand that your mileage may vary between companies, teams and project and study results represent an *average* result. Studies which have inconclusive data either because results vary too greatly simply have a null finding and conclude that you can't conclude anything for sure or that more study is required. This finding in and of itself is valuable because it would imply that DevOps may not actually be beneficial and it *isn't* a sure bet for improvement.  
  
  ---
  
  