Answers: <p>The key metric for a DevOps pipeline is <strong>Cycle Time</strong> (also called <strong>Lead Time</strong>).  This is the time it takes for a change (or a change request, tracking all the way to the inception of the idea).  The best illustration of this concept I know of is from the book "The Goal", which talks about it in a manufacturing context.</p>
  
  <p><strong>Deployment Frequency</strong> is useful as well.  We want deployments to be frequent in a DevOps pipeline.  There is no magical "1 day is good, 2 days is bad" measurement; this will need a historical context on your project to be meaningful.</p>
  
  <p><strong>Deployment Size</strong>: Measured in however your developers measure work - user stories, story points, quatloos, whatever.  Again, you want to see trends over time, not absolute value.</p>
  
  <p>Between frequency and size there is a story to tell.  Are our releases becoming more infrequent and larger?  why?  Are they becoming smaller and more frequent?  Again, why?</p>
  
  <p>Explaining whether the frequency/size trend is good, we will also need <strong>Percentage of Failed Deploys</strong>.  Uncovering the 'why' in those three metrics will tell you a lot about the health of the project.</p>
  
  <p>My personal favorite, although it is a vanity metric, is <strong>Time for a Trivial Deploy</strong>.  If you found the smallest possible thing worth redeploying the entire site over... perhaps a typo in the CEO's name... how quickly could you go from the panic phone call to a deployed site?  I say 'vanity' because it really isn't that predictive beyond what the other metrics above discuss, but makes me feel good when I like the value.</p>
  
  <p>If we get into monitoring, there are a bunch of different things we can track...  from all-encompassing things like '<strong>Uptime</strong>', to really low level things like the <strong>time spent regenerating custom HTML</strong> on a request-response cycle...  but those aren't specific to instituting a DevOps culture.</p>
  
  <p>These don't directly tie to dollars... doing so will require more knowledge about your org than I can offer in a forum like this;  but they are the key to BEGIN to answer that question.  Once you know you are able to regularly release work into production as a non-event, you can begin to see how much effort you were wasting before.  As the book "The Goal" teaches (about manufacturing pipelines - it's relevant), optimizing locally can <em>look</em> like you're saving money, but ultimately, it just creates value that is tied up in inventory (undeployed features).</p>
  
  <p>Beyond this advice, you should take a look at the <a href="https://puppet.com/resources/whitepaper/2016-state-of-devops-report" rel="noreferrer">State Of DevOps Report</a> for the past few years.  This is full of measurements about real world projects that you could emulate.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-07T22:31:15.693Z
Answers: 2017-03-07T22:36:56.543Z
Answers: 2017-03-07T22:36:56.543Z
Answers: David Bock
Answers: <p>Captain obvious : by reducing time to market and defects on releases.</p>
  
  <p>To extend on this one liner, the usual pitfall is being an organization change without any reference. </p>
  
  <p>Engaging a culture or organization change implies some expense to train and introduce people to this new method, this have a cost in training but also imply a loss in productivity as people in a train session won't produce anything. This is the investment part of the cultural change.</p>
  
  <p>To measure a ROI you have to first find some relevant metrics which should be improved (understand costly, either expensive or source of loss of profit). This could be a shorter time to market, less patch after each release, a better customer engagement within your product. Relevant metrics will be highly dependent of your product(s).</p>
  
  <p>Measuring a ROI (how fast you have covered the training expense) implies that you can factually present an evolution on those metrics, so before engaging any change you must have defined those metrics and measured them in an objective way.<br>
  Once you have a real evolution to show you can tell if you did improve something in a way which has covered the training expense and become more profitable than it was before.</p>
  
  <p>The usual pitfall is to engage the change before having defined any metrics and thus evaluating the ROI on a feeling and not on factual datas.</p>
  
  <p>Productivity can be a metric, but its measurement is usually very hard to do in an objective fashion and should not be a first class metric for this kind of study.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-07T22:01:37.137Z
Answers: 2017-03-08T08:10:51.310Z
Answers: 2017-03-08T08:10:51.310Z
Answers: Tensibai
Answers: <p>Late to the game here but thought I'd chime in.</p>
  
  <p>You can't manage what you don't measure, so for starters, here are the key metrics devops teams should be tracking for incident response:</p>
  
  <ul>
  <li><strong>Uptime %</strong> : total % of time available = [total time - downtime] / [total time]</li>
  <li><strong>MTTR</strong> : mean time to resolution = [downtime] / [# incidents]</li>
  <li><strong>MTTA</strong> : mean time to acknowledge = [total time to acknowledge] / [# of incidents]</li>
  <li><strong>MTBF</strong> : mean time between failures = [total time - downtime] / [# of incidents]</li>
  </ul>
  
  <p>These metrics give you a high level health check of your operations, and help you identify where you need to dig in further.</p>
  
  <p>Take a look at the <a href="https://pagertree.com/2018/11/20/what-is-mttr/" rel="nofollow noreferrer">whiteboard animation here</a> for a more in-depth look on the topic.</p>
  
  <p>Once you know your metrics, you can bump them up against the <a href="https://www.itondemand.com/2018/05/29/costs-of-downtime/" rel="nofollow noreferrer">cost of downtime</a>. You can start building out your team's ROI from there and set some quantitative metrics for continuous improvement.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-12-03T01:50:53.957Z
Answers: Those are reliability metrics, which are related to DevOps but aren't enough. Reliability is only one aspect of DevOps.
Answers: MisterStrickland
Answers: CC BY-SA 4.0
Answers: 2018-12-05T05:47:10.857Z
Answers: Thanks @Phil. That's a good point - these are indeed metrics focused on reliability, which is an important part of DevOps, but certainly not the whole thing. Hopefully staying on top of reliability is a good starting point, but don't stop there!
Answers: Yuan Cheng
Answers: CC BY-SA 4.0
Answers: 2018-12-01T01:40:48.667Z
Answers: 2018-12-01T01:40:48.667Z
Answers: Yuan Cheng
Answers: <p>Great question! Most of us know investing in DevOps practices is a highly worthwhile pursuit for myriad reasons; we don't often justify DevOps on the impact to the bottom line alone, though.</p>
  
  <p><strong>Note</strong>: this is something of an opinionated question, and my answer is likewise, opinionated.</p>
  
  <p>Tensibai wisely suggested we find the right metrics, and he used time-to-market as an example. This is a great big-picture approach.</p>
  
  <p>As an alternative approach, my experience with bean-counters is that they don't need to--or necessarily want to--know the big-picture, they just want evidence of fiscal responsibility. They want to see a trend in the right direction.</p>
  
  <p>Here are just a few fiscal wins:</p>
  
  <ul>
  <li>calculate the server costs saved by leveraging <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/auto-scaling-benefits.html" rel="noreferrer">auto-scaling</a> in the cloud</li>
  <li>for income-generating sites, extrapolate the cost-per-minute of downtime, then show improvements in <a href="https://www.sumologic.com/what-are-mttr-and-mtti/" rel="noreferrer">MTTI and MTTR</a></li>
  <li>again, for income-generating sites, estimate the cost-per-minute saved by leveraging highly-available architecture based on past incidents</li>
  <li>if you've improved on your build and deploy pipeline, show that you've reduced regressions and errors in production caused by already-tracked faults</li>
  <li>if you've made improvements to developer test environments, or even the tools and configuration on developer laptops, look at commit histories to see if new engineers are making their first contributions sooner after joining</li>
  <li>perform a full cost comparison between cloud and on-prem, much like <a href="https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-the-cloud/" rel="noreferrer">Gitlab did recently</a>, to justify your infrastructure spending (a.k.a. savings!)</li>
  </ul>
  
  <p>Showing that you're money-conscious and you have a few clear wins is often sufficient. I've certainly missed some obvious examples; feel free to add comments below.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-09T20:48:20.290Z
Answers: I'm 1000% behind this. I think we're on the cusp of a big shift in monitoring: I no longer care about how much CPU or RAM is in use on any given instance, I care about how much I'm paying for an auto-scaling group of instances over time. I don't care how many requests an instance can handle, I care about how much it costs to service a request. This shift in thinking can really help drive better metrics for DevOps, including ROI.
Answers: Adrian
Answers: CC BY-SA 3.0
Answers: 2017-03-08T00:04:08.010Z
Answers: 2017-03-08T00:04:08.010Z
Answers: Woodland
Body: <p>DevOps is complex, and involves many non-deterministic aspects like culture and process.<br>
  What are some ways to measure DevOps initiatives for success?<br>
  How do you prove to a business that the investment they have made is returning (or saving) real dollars?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-08T09:59:11.727Z
Comments: Hey Dave, I wonder what "you" mean by "measurement", as per one of the tags you used in your question. IMO (not more then that), only using the (existing) "metrics" tag would be more appropriate. No? If you don't agree (which is ok ...), would you mind (briefy) explaining what you think the difference between those 2 tags is (or should be)?
Comments: Pierre.Vriens
Comments: CC BY-SA 3.0
Comments: 2017-03-09T15:51:19.567Z
Comments: @Pierre.Vriens I suppose measurement is the act of gathering data, whereas a metric is something you measure.  I removed the "measurement" tag because it's probably redundant.
Comments: Dave Swersky
Tags: <metrics><roi>
Title: What are some methods to measure the ROI for DevOps?
UserDisplayName: Dave Swersky
ThreadText: # What are some methods to measure the ROI for DevOps?
  
  (Asked by: username_0 on 2017-03-07)
  
  DevOps is complex, and involves many non-deterministic aspects like culture and process.  
  What are some ways to measure DevOps initiatives for success?  
  How do you prove to a business that the investment they have made is returning (or saving) real dollars?
  
  **Tags:** metrics, roi
  
  > **Comments:**
  > **username_1**: Hey username_0, I wonder what "you" mean by "measurement", as per one of the tags you used in your question. IMO (not more then that), only using the (existing) "metrics" tag would be more appropriate. No? If you don't agree (which is ok ...), would you mind (briefy) explaining what you think the difference between those 2 tags is (or should be)?  
  > **username_0**: @username_1 I suppose measurement is the act of gathering data, whereas a metric is something you measure.  I removed the "measurement" tag because it's probably redundant.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_6 on 2017-03-08. Score: 17 | Accepted answer
  
  Great question! Most of us know investing in DevOps practices is a highly worthwhile pursuit for myriad reasons; we don't often justify DevOps on the impact to the bottom line alone, though.
  
  **Note**: this is something of an opinionated question, and my answer is likewise, opinionated.
  
  username_3 wisely suggested we find the right metrics, and he used time-to-market as an example. This is a great big-picture approach.
  
  As an alternative approach, my experience with bean-counters is that they don't need to--or necessarily want to--know the big-picture, they just want evidence of fiscal responsibility. They want to see a trend in the right direction.
  
  Here are just a few fiscal wins:
  
  * calculate the server costs saved by leveraging [auto-scaling](http://docs.aws.amazon.com/autoscaling/latest/userguide/auto-scaling-benefits.html) in the cloud
  * for income-generating sites, extrapolate the cost-per-minute of downtime, then show improvements in [MTTI and MTTR](https://www.sumologic.com/what-are-mttr-and-mtti/)
  * again, for income-generating sites, estimate the cost-per-minute saved by leveraging highly-available architecture based on past incidents
  * if you've improved on your build and deploy pipeline, show that you've reduced regressions and errors in production caused by already-tracked faults
  * if you've made improvements to developer test environments, or even the tools and configuration on developer laptops, look at commit histories to see if new engineers are making their first contributions sooner after joining
  * perform a full cost comparison between cloud and on-prem, much like [Gitlab did recently](https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-the-cloud/), to justify your infrastructure spending (a.k.a. savings!)
  
  Showing that you're money-conscious and you have a few clear wins is often sufficient. I've certainly missed some obvious examples; feel free to add comments below.
  
  > **Comments:**
  > **username_7**: I'm 1000% behind this. I think we're on the cusp of a big shift in monitoring: I no longer care about how much CPU or RAM is in use on any given instance, I care about how much I'm paying for an auto-scaling group of instances over time. I don't care how many requests an instance can handle, I care about how much it costs to service a request. This shift in thinking can really help drive better metrics for DevOps, including ROI.  
  
  ---
  
  ### Answer by username_2 on 2017-03-07. Score: 12
  
  The key metric for a DevOps pipeline is **Cycle Time** (also called **Lead Time**). This is the time it takes for a change (or a change request, tracking all the way to the inception of the idea). The best illustration of this concept I know of is from the book "The Goal", which talks about it in a manufacturing context.
  
  **Deployment Frequency** is useful as well. We want deployments to be frequent in a DevOps pipeline. There is no magical "1 day is good, 2 days is bad" measurement; this will need a historical context on your project to be meaningful.
  
  **Deployment Size**: Measured in however your developers measure work - user stories, story points, quatloos, whatever. Again, you want to see trends over time, not absolute value.
  
  Between frequency and size there is a story to tell. Are our releases becoming more infrequent and larger? why? Are they becoming smaller and more frequent? Again, why?
  
  Explaining whether the frequency/size trend is good, we will also need **Percentage of Failed Deploys**. Uncovering the 'why' in those three metrics will tell you a lot about the health of the project.
  
  My personal favorite, although it is a vanity metric, is **Time for a Trivial Deploy**. If you found the smallest possible thing worth redeploying the entire site over... perhaps a typo in the CEO's name... how quickly could you go from the panic phone call to a deployed site? I say 'vanity' because it really isn't that predictive beyond what the other metrics above discuss, but makes me feel good when I like the value.
  
  If we get into monitoring, there are a bunch of different things we can track... from all-encompassing things like '**Uptime**', to really low level things like the **time spent regenerating custom HTML** on a request-response cycle... but those aren't specific to instituting a DevOps culture.
  
  These don't directly tie to dollars... doing so will require more knowledge about your org than I can offer in a forum like this; but they are the key to BEGIN to answer that question. Once you know you are able to regularly release work into production as a non-event, you can begin to see how much effort you were wasting before. As the book "The Goal" teaches (about manufacturing pipelines - it's relevant), optimizing locally can *look* like you're saving money, but ultimately, it just creates value that is tied up in inventory (undeployed features).
  
  Beyond this advice, you should take a look at the [State Of DevOps Report](https://puppet.com/resources/whitepaper/2016-state-of-devops-report) for the past few years. This is full of measurements about real world projects that you could emulate.
  
  
  ---
  
  ### Answer by username_3 on 2017-03-07. Score: 8
  
  Captain obvious : by reducing time to market and defects on releases.
  
  To extend on this one liner, the usual pitfall is being an organization change without any reference.
  
  Engaging a culture or organization change implies some expense to train and introduce people to this new method, this have a cost in training but also imply a loss in productivity as people in a train session won't produce anything. This is the investment part of the cultural change.
  
  To measure a ROI you have to first find some relevant metrics which should be improved (understand costly, either expensive or source of loss of profit). This could be a shorter time to market, less patch after each release, a better customer engagement within your product. Relevant metrics will be highly dependent of your product(s).
  
  Measuring a ROI (how fast you have covered the training expense) implies that you can factually present an evolution on those metrics, so before engaging any change you must have defined those metrics and measured them in an objective way.  
  Once you have a real evolution to show you can tell if you did improve something in a way which has covered the training expense and become more profitable than it was before.
  
  The usual pitfall is to engage the change before having defined any metrics and thus evaluating the ROI on a feeling and not on factual datas.
  
  Productivity can be a metric, but its measurement is usually very hard to do in an objective fashion and should not be a first class metric for this kind of study.
  
  
  ---
  
  ### Answer by username_4 on 2018-12-01. Score: 1
  
  Late to the game here but thought I'd chime in.
  
  You can't manage what you don't measure, so for starters, here are the key metrics devops teams should be tracking for incident response:
  
  * **Uptime %** : total % of time available = [total time - downtime] / [total time]
  * **MTTR** : mean time to resolution = [downtime] / [# incidents]
  * **MTTA** : mean time to acknowledge = [total time to acknowledge] / [# of incidents]
  * **MTBF** : mean time between failures = [total time - downtime] / [# of incidents]
  
  These metrics give you a high level health check of your operations, and help you identify where you need to dig in further.
  
  Take a look at the [whiteboard animation here](https://pagertree.com/2018/11/20/what-is-mttr/) for a more in-depth look on the topic.
  
  Once you know your metrics, you can bump them up against the [cost of downtime](https://www.itondemand.com/2018/05/29/costs-of-downtime/). You can start building out your team's ROI from there and set some quantitative metrics for continuous improvement.
  
  > **Comments:**
  > **username_5**: Those are reliability metrics, which are related to DevOps but aren't enough. Reliability is only one aspect of DevOps.  
  > **username_4**: Thanks @Phil. That's a good point - these are indeed metrics focused on reliability, which is an important part of DevOps, but certainly not the whole thing. Hopefully staying on top of reliability is a good starting point, but don't stop there!  
  
  ---
  
  