Answers: <p>CloudWatch is absolutely <a href="https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/" rel="noreferrer">the way to go</a> on this front, you can surface Kafka Metrics in CloudWatch either by:</p>
  
  <ol>
  <li>Having a <a href="https://github.com/sidcarter/kafka-cloudwatch" rel="noreferrer">separate process</a> that pushes the metrics into CloudWatch.</li>
  <li>Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.</li>
  </ol>
  
  <p>The process you follow will broadly be broken down into:</p>
  
  <ol>
  <li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html" rel="noreferrer">Publish Custom Metrics</a>.</li>
  <li>Create an <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-lc" rel="noreferrer">AutoScaling Launch Configuration</a>.</li>
  <li>Create an <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-asg" rel="noreferrer">AutoScaling Group and Policies</a> based upon your Custom Metric.</li>
  </ol>
  
  <p>Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use <a href="https://aws.amazon.com/ec2/spot/" rel="noreferrer">Spot Instances</a> as part of your autoscaling configuration.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-11-08T17:30:57.123Z
Answers: The first link is dead
Answers: Marquis Blount
Answers: CC BY-SA 3.0
Answers: 2017-12-14T00:12:19.073Z
Answers: @MarquisBlount here's the [Google Cache for the page](https://webcache.googleusercontent.com/search?q=cache:mM9WvsLriCQJ:https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/)
Answers: dededecline
Answers: CC BY-SA 3.0
Answers: 2017-04-04T14:09:47.813Z
Answers: 2017-04-04T14:09:47.813Z
Answers: Richard Slater
Body: <p>I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.</p>
  
  <p>Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?</p>
  
  <p>Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-04-04T12:53:19.453Z
Comments: Maybe you can explain a little more why you think cloudwatch is not a good approach ?
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-04-04T14:05:09.723Z
Comments: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.
Comments: dededecline
Tags: <jenkins><amazon-web-services><autoscaling>
Title: How can I autoscale instances based on Kafka topic lag?
UserDisplayName: dededecline
ThreadText: # How can I autoscale instances based on Kafka topic lag?
  
  (Asked by: username_0 on 2017-04-03)
  
  I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.
  
  Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?
  
  Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.
  
  **Tags:** jenkins, amazon-web-services, autoscaling
  
  > **Comments:**
  > **username_1**: Maybe you can explain a little more why you think cloudwatch is not a good approach ?  
  > **username_0**: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-04-04. Score: 7 | Accepted answer
  
  CloudWatch is absolutely [the way to go](https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/) on this front, you can surface Kafka Metrics in CloudWatch either by:
  
  1. Having a [separate process](https://github.com/sidcarter/kafka-cloudwatch) that pushes the metrics into CloudWatch.
  2. Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.
  
  The process you follow will broadly be broken down into:
  
  1. [Publish Custom Metrics](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html).
  2. Create an [AutoScaling Launch Configuration](http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-lc).
  3. Create an [AutoScaling Group and Policies](http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-asg) based upon your Custom Metric.
  
  Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use [Spot Instances](https://aws.amazon.com/ec2/spot/) as part of your autoscaling configuration.
  
  > **Comments:**
  > **username_3**: The first link is dead  
  > **username_0**: @username_3 here's the [Google Cache for the page](https://webcache.googleusercontent.com/search?q=cache:mM9WvsLriCQJ:https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/)  
  
  ---
  
  