Answers: <p>Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain.</p>
  
  <p>As such I've turn toward <a href="https://prometheus.io/" rel="noreferrer">prometeus</a> which works with a simple contract, an exporter does the collection job (could be the <a href="https://github.com/prometheus/cloudwatch_exporter" rel="noreferrer">cloudwatch_exporter</a> on AWS or when using <a href="https://github.com/firehol/netdata" rel="noreferrer">netdata</a> querying it (either a central one or each separately, but the latter lost the advantage of auto-discovery) (doc for the netdata part <a href="https://github.com/firehol/netdata/wiki/Using-Netdata-with-Prometheus" rel="noreferrer">here</a>) and there's also a node exporter from prometeus team.</p>
  
  <p>The main advantage behind prometheus is the use of <a href="https://prometheus.io/docs/querying/functions/" rel="noreferrer">mathematics functions</a> and grouping in the alert definition by label from the metric name. This allow to define alerts on various way, for my infrastructure monitoring I work with standard deviation or standard variance to avoid fixed level of alert on cpu usage, it alerts me if the stddev goes up by 20%.</p>
  
  <p>So for your case you could export your database or push the metrics to prometeus and assuming a metric name <code>client_queries</code> with the client name as the label <code>client</code> you could do something along the line of this (untested of course and just to illustrate):</p>
  
  <pre><code>ALERT ClientQueryDriftUp
    IF avg(client_queries[5m]) &gt; avg(client_queries[5m] offset 1w) * 1.2
    ANNOTATIONS {
      summary = "{{ $labels.client }} query has gone up over 20% of last week",
      description = "{{ $labels.client}} query average is high ! (current average value: {{ $value }}s)",
    }
  </code></pre>
  
  <p>Or if you just want an alert if the difference between averages is over 1 second (assuming values are in milliseconds) you can use this IF condition:</p>
  
  <pre><code>abs(avg(client_queries[5m]) - avg(client_queries[5m] offset 1w)) &gt; 1000
  </code></pre>
  
  <p>Of course this answer is just an overview on how it can tackle your problem and is far from exhaustive about it.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-08-29T08:45:33.723Z
Answers: 2017-08-29T08:45:33.723Z
Answers: Tensibai
Answers: <p>I would suggest <a href="https://github.com/Yelp/elastalert" rel="nofollow noreferrer">elastalert</a> as an alert system for elasticsearch, it has amazing capabilities, you can execute any query on elasticsearch and define an alarm depending on the response and it support multi ways for alert like sending email or slack or telegram, even execute specific commands.</p>
  
  <p>For monitor data in elasticsearch as you are already using ELK, you can use Kibana for monitoring not only for making elasticsearch query and create very valuable Dashboards</p>
  
  <p>For hardware and services I think Nagios do very good job there, but for more easy solution you can use <a href="http://www.monitis.com/server-monitoring" rel="nofollow noreferrer">Monit</a> it's very easy to use and configure and can do a lot of jobs and service management, But the same as you are already using ELK, you can configure <a href="https://www.elastic.co/downloads/beats/metricbeat" rel="nofollow noreferrer">metricbeat</a> to monitor the server (CPU, RAM, Disk usage, etc...)</p>
  
  <p>For me I think the best thing is to use ELK with all the integration that you need, that will allow you to more focus on, and centralize your monitoring and alerting systems  </p>
  
Answers: CC BY-SA 3.0
Answers: 2017-08-29T01:32:53.067Z
Answers: 2017-08-29T01:32:53.067Z
Answers: Wissam Roujoulah
Answers: <p>We use Datadog's anomaly detection for this purpose:</p>
  
  <p><a href="https://www.datadoghq.com/blog/introducing-anomaly-detection-datadog/" rel="nofollow noreferrer">https://www.datadoghq.com/blog/introducing-anomaly-detection-datadog/</a></p>
  
  <p>We push custom metrics which in your case would be something like query execution time tagged by client name, and you can create a single Multi monitor that would alert based on anomalous behavior on this metric.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-08-29T17:40:19.333Z
Answers: 2017-08-29T17:40:19.333Z
Answers: Olivier Le Floch
Answers: <p>Consider using something like <a href="https://www.nagios.org/" rel="noreferrer">nagios</a> or <a href="https://www.zabbix.com/" rel="noreferrer">Zabbix</a>. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way to avoid this custom monitoring. While monitoring, alerting and reporting management and configuration can be tedious, there is no easy shortcut for doing it properly and it's probably going to take more time, dedication and resources to do it properly than you think.</p>
  
  <p>Out of the 4 monitoring systems I have used extensively, I must recommend nagios - it seems to be the best suited for configuration management systems and most compatable with DevOps philosophy.</p>
  
  <p>It may sometimes be useful to use a system such as elastalert or similar systems as middleware for alerting/monitoring/reporting, but you will have to judge each situation/monitor on a case-by-case basis, but you will find these tools (and most monitoring systems) have some SDK, API or tool capable of helping you with your monitoring customization. Just choose the easiest to configure system with the lowest ROI and support cost and be sure to stick with it and follow your monitoring migration project to completion. Be sure to consider dedicated resources/personnel for this task if possible. </p>
  
Answers: CC BY-SA 3.0
Answers: 2017-08-29T03:06:50.150Z
Answers: 2017-08-29T03:06:50.150Z
Answers: James Shewey
Body: <p>I am looking for a monitoring/alerting tool/system which can support very arbitrary data source and logic.</p>
  
  <p>(following is a made up example to more easily demonstrate my needs)</p>
  
  <p>I have a database where I can find out the query execution times for my clients(which are individual companies).  The query times are expected to vary greatly between clients, because of the data of each company.  I want to have an alert when the query time for a client is ?% over some median value from the past, and where I am comparing Monday's data against previous Mondays' data(I am being difficult for demonstration purpose).</p>
  
  <p>I want to avoid defining a separate alert for each company client.</p>
  
  <p>I want to avoid defining a separate alert for each kind of query I want to monitor.</p>
  
  <p>I want to avoid having each source pre-push data to one central place.</p>
  
  <p>I would like to monitor some other things that has nothing to do with databases, and want to apply some entirely different logic for my alerts - e.g. some REST api call to some other system, or some ELK or time series data, or some hardware monitoring data.</p>
  
  <p>I am contemplating a custom solution, but want to see if there is already a solution out there that can meet some or more of my needs.  I have seen/used some monitoring solutions out there such as Nagios, though not an expert on those.  Will appreciate any pointers or recommendation.</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-08-29T07:59:01.233Z
Comments: You may have a look at [prometheus](https://prometheus.io/), it has the ability to alert on standard deviation, you'll still have to create an exporter for the datas, but that's a matter of doing your queries and returning the list accordingly, prometeus can then collect from this list and will discover new entries by itself.
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-08-29T09:12:38.807Z
Comments: TICK stack is really good. Actually, T I G(grafana) stack is much better.
Comments: luv.preet
Tags: <monitoring>
Title: monitoring/alerting system which can support very arbitrary data source and logic
UserDisplayName: Fai Ng
ThreadText: # monitoring/alerting system which can support very arbitrary data source and logic
  
  (Asked by: username_0 on 2017-08-29)
  
  I am looking for a monitoring/alerting tool/system which can support very arbitrary data source and logic.
  
  (following is a made up example to more easily demonstrate my needs)
  
  I have a database where I can find out the query execution times for my clients(which are individual companies). The query times are expected to vary greatly between clients, because of the data of each company. I want to have an alert when the query time for a client is ?% over some median value from the past, and where I am comparing Monday's data against previous Mondays' data(I am being difficult for demonstration purpose).
  
  I want to avoid defining a separate alert for each company client.
  
  I want to avoid defining a separate alert for each kind of query I want to monitor.
  
  I want to avoid having each source pre-push data to one central place.
  
  I would like to monitor some other things that has nothing to do with databases, and want to apply some entirely different logic for my alerts - e.g. some REST api call to some other system, or some ELK or time series data, or some hardware monitoring data.
  
  I am contemplating a custom solution, but want to see if there is already a solution out there that can meet some or more of my needs. I have seen/used some monitoring solutions out there such as Nagios, though not an expert on those. Will appreciate any pointers or recommendation.
  
  **Tags:** monitoring
  
  > **Comments:**
  > **username_1**: You may have a look at [prometheus](https://prometheus.io/), it has the ability to alert on standard deviation, you'll still have to create an exporter for the datas, but that's a matter of doing your queries and returning the list accordingly, prometeus can then collect from this list and will discover new entries by itself.  
  > **username_2**: TICK stack is really good. Actually, T I G(grafana) stack is much better.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2017-08-29. Score: 6 | Accepted answer
  
  Consider using something like [nagios](https://www.nagios.org/) or [Zabbix](https://www.zabbix.com/). While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way to avoid this custom monitoring. While monitoring, alerting and reporting management and configuration can be tedious, there is no easy shortcut for doing it properly and it's probably going to take more time, dedication and resources to do it properly than you think.
  
  Out of the 4 monitoring systems I have used extensively, I must recommend nagios - it seems to be the best suited for configuration management systems and most compatable with DevOps philosophy.
  
  It may sometimes be useful to use a system such as elastalert or similar systems as middleware for alerting/monitoring/reporting, but you will have to judge each situation/monitor on a case-by-case basis, but you will find these tools (and most monitoring systems) have some SDK, API or tool capable of helping you with your monitoring customization. Just choose the easiest to configure system with the lowest ROI and support cost and be sure to stick with it and follow your monitoring migration project to completion. Be sure to consider dedicated resources/personnel for this task if possible.
  
  
  ---
  
  ### Answer by username_1 on 2017-08-29. Score: 5
  
  Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain.
  
  As such I've turn toward [prometeus](https://prometheus.io/) which works with a simple contract, an exporter does the collection job (could be the [cloudwatch\_exporter](https://github.com/prometheus/cloudwatch_exporter) on AWS or when using [netdata](https://github.com/firehol/netdata) querying it (either a central one or each separately, but the latter lost the advantage of auto-discovery) (doc for the netdata part [here](https://github.com/firehol/netdata/wiki/Using-Netdata-with-Prometheus)) and there's also a node exporter from prometeus team.
  
  The main advantage behind prometheus is the use of [mathematics functions](https://prometheus.io/docs/querying/functions/) and grouping in the alert definition by label from the metric name. This allow to define alerts on various way, for my infrastructure monitoring I work with standard deviation or standard variance to avoid fixed level of alert on cpu usage, it alerts me if the stddev goes up by 20%.
  
  So for your case you could export your database or push the metrics to prometeus and assuming a metric name `client_queries` with the client name as the label `client` you could do something along the line of this (untested of course and just to illustrate):
  
  ```
  ALERT ClientQueryDriftUp
    IF avg(client_queries[5m]) > avg(client_queries[5m] offset 1w) * 1.2
    ANNOTATIONS {
      summary = "{{ $labels.client }} query has gone up over 20% of last week",
      description = "{{ $labels.client}} query average is high ! (current average value: {{ $value }}s)",
    }
  
  ```
  
  Or if you just want an alert if the difference between averages is over 1 second (assuming values are in milliseconds) you can use this IF condition:
  
  ```
  abs(avg(client_queries[5m]) - avg(client_queries[5m] offset 1w)) > 1000
  
  ```
  
  Of course this answer is just an overview on how it can tackle your problem and is far from exhaustive about it.
  
  
  ---
  
  ### Answer by username_3 on 2017-08-29. Score: 3
  
  I would suggest [elastalert](https://github.com/Yelp/elastalert) as an alert system for elasticsearch, it has amazing capabilities, you can execute any query on elasticsearch and define an alarm depending on the response and it support multi ways for alert like sending email or slack or telegram, even execute specific commands.
  
  For monitor data in elasticsearch as you are already using ELK, you can use Kibana for monitoring not only for making elasticsearch query and create very valuable Dashboards
  
  For hardware and services I think Nagios do very good job there, but for more easy solution you can use [Monit](http://www.monitis.com/server-monitoring) it's very easy to use and configure and can do a lot of jobs and service management, But the same as you are already using ELK, you can configure [metricbeat](https://www.elastic.co/downloads/beats/metricbeat) to monitor the server (CPU, RAM, Disk usage, etc...)
  
  For me I think the best thing is to use ELK with all the integration that you need, that will allow you to more focus on, and centralize your monitoring and alerting systems
  
  
  ---
  
  ### Answer by username_4 on 2017-08-29. Score: 2
  
  We use Datadog's anomaly detection for this purpose:
  
  <https://www.datadoghq.com/blog/introducing-anomaly-detection-datadog/>
  
  We push custom metrics which in your case would be something like query execution time tagged by client name, and you can create a single Multi monitor that would alert based on anomalous behavior on this metric.
  
  
  ---
  
  