Understanding How To Monitor a Spark Cluster
Problem Statement

Building a spark cluster is easy, however, one needs to also monitor the overall cluster routinely to make sure that whatever load estimations and assumptions we have made while creating the cluster holds true and the cluster doesn’t start giving us surprises.

Also, once we have our cluster deployed, we generally start adding more and more load to it.

Another factor that can affect how a cluster behaves is the concurrent workload. As time passes, upstream data availability patterns and data sizes may change. If earlier 5 heavy & 10 medium sized jobs were running at 7AM morning, now, 3 months down the line we can have a situation where 10 heavy & 5 medium jobs are running at 7AM. The concurrency remains same but overall workload changes.

Hence it becomes important to monitor your cluster periodically along with setting up alerting.

Pre-Requisites

Basics of Spark Basics of Hadoop

Cluster/EMR Jargon

Spot Nodes — It is a general design pattern (to keep costs under control & take advantage of cloud environment) that most EMR clusters are configured with some fixed capacity (on-demand) machines and some variable capacity (Spot Instances). These spot instances are around 70% cheaper than on-demand instances as it utilizes unused capacity of the cloud provider.

However, as a downside, these spot instances can be taken away by cloud provider to manage high demand of on-demand instances across the cloud.

Cluster Scaling (Up & Down) — The EMR or any cloud based Hadoop cluster can scale up (i.e. increase its available resources) as and when a high load comes.

Similarly, the cluster can also release resources once the load decreases. It is usually a configuration setting that decides the scale up & down limits along with the strategy used to scale up or down. This is one of the advantages of being on cloud vs on-premises setup that you can scale up & down ,mostly, as per your demand.

Master Node — It is used mainly for maintenance and up-keeping of EMR

Core Nodes — They are used to host HDFS and run Spark application’s driver.

Task Nodes — They are primarily ment to run Spark Application’s executors. However, they also host Spark Driver (Configurable property).

EMRFS — A file output committer designed by AWS specifically for EMR. It has an advantage over open source file committer as EMRFS is optimized for use with cloud storages like S3 which OSS file committer is not. EMRFS syncs files to S3 in parallel and handles file renames in a better way.

Important Monitoring Metrics

Cluster Monitoring Dashboard

Allocated vs Used Memory

Last widget, used to see how much resources are available in cluster and how is the cluster scaling up & down.

Running jobs

How many jobs are in running state. Most times, we see that a lot of flows are running on scheduler and most of them have some tasks in re-try/waiting state due to dependency. However, we can check how many jobs are executing on Cluster via this widget.

Pending Jobs

A spike in this widget signifies that the cluster is not able to handle the current job load.

If Allocated memory is also less then this can be due to cluster scaling issues, However, on a bad day, there can be a case that a lot of upstream tables are late and now a lot of job wants to run at the same time. Hence the spike.

HDFS Available (or Local Disk Used)

HDFS is formed from the locally attached HDD of core nodes in the EMR (Task nodes local HDD is not part of HDFS). When a spark job flushes the output to the S3, it stores the output in local HDD storage and then flushes it to S3. Each such incident is marked by a decline in available HDFS %.

It can be a cause of concern if on someday, we observe that available HDFS % declines to seriously low levels, like 5–10% (or in case of Disk Usage, it spikes to 90–95%).

Container failures

If we see a lot of container failures happening on a particular day, then this will lead to spark applications retrying a lot of failed tasks. We can experience poor spark job performance on and after the time container failures happen. Routine spot node leaving the cluster are (most probably) not classified as container failures.

Master Nodes Available

The number of master nodes active in the system. We will see this count come down if the cluster is experiencing a master node failure.

Core nodes Available

Master nodes are only responsible for up-keeping and maintenance of cluster. However, the core nodes are used to run spark application driver & hosts HDFS.

Core nodes going down can significantly reduce the concurrent job execution capabilities of the cluster.

Other Useful Widgets:

CPU Usage (Master & Core nodes) : Instance can fail if it CPU usage is routinely spiking to +90%

Memory Usage (Master & Core nodes): Similarly for memory usage.

Disk IOPS: Consistently high disk IOPS can signify cluster choking due to a lot of concurrent write & reads from Disk. Probable solution can be increasing the provisioned IOPS of disks attached to task & core nodes.

A few spikes here and there should be fine for this metric.

Cluster Metrics Collection Pipeline

Services running on a cluster usually emits metrics on their respective ports and any metric collection agent configured to collect metrics from those ports can collect these metrics.

A metrics collection agent like Telegraf can be installed on all nodes of EMR/Cluster (Option available in EMR Bootstrap actions, or in custom OS image used to create nodes of Hadoop cluster) and it can be configured to collect different metrics from different ports.

Telegraf then sends the metrics to the configured Prometheus URL.

Metrics Collection Pipeline

Grafana dashboards can be setup on top of Prometheus to show similar metrics & widgets as displayed above.

Alerting

We can implement multiple anomaly or threshold based alerts over these metrics using Prometheus.

Sample query of creating an alert on “Apps Running” greater than 20 by querying Prometheus will look something like this

avg(topk(1, Hadoop_ResourceManager_AppsRunning{role=”emr”,project=”your_project”}[10m])) > 20

Closing Notes

There is no one right strategy to monitor your cluster.

As a practice, most system owners routinely monitor their system, along with setting hard alerts, till the time system becomes a bit mature & stable.

I hope the above mentioned metrics will guide you in taking your cluster in the direction of stability.

Happy learning!!
['Parth Gupta']
['Spark', 'Alerting', 'Cluster', 'Monitoring', 'Metrics']