We've written before about how easy it is to spin up an Elasticsearch cluster on Azure, using our offering on the Azure Marketplace. The Marketplace is a fantastic way to try Elasticsearch on Azure since all the hard work of provisioning resources and wiring them together is taken care of, leaving you just the simple task of providing the inputs to the step-by-step wizard. A few minutes later and Voila! you have a cluster!
Once trying the Marketplace offering however, you're most likely going to want to integrate deployments into your build automation pipeline of choice, and this is where the ARM template in our github repository comes in; using the Azure CLI or PowerShell SDKs, it's easy to target a specific release of the template for reliable deployments. What's more, the template accepts many additional parameters to configure features such as snapshot and restore to an Azure storage account, Transport Layer Security, load balancing and SSL offload with Application Gateway, as well as being able to configure the number and size of disks to attach to each data node. The latter is particularly useful when you have a relatively small data set, so your storage demands are not large, but require virtual machines with sufficient RAM and CPU for your running workload. For this post, let's explore a few different scenarios.
These options are available to you whether you're running on Windows, Linux or OSX. Cross-platform FTW!
Now we're ready to automate!
Note that the release versions of the ARM template do not necessarily align with the version of Elasticsearch and Kibana deployed by default; the template is able to deploy many different versions of Elasticsearch and Kibana, which can be controlled through the esVersion parameter, deploying the latest version of Elasticsearch and Kibana if unspecified. For template version 5.1.2, this is Elasticsearch and Kibana 5.3.1. We're looking to align the template version with the Elastic stack version in a future release.
The 5.1.2 tag release deploys Elasticsearch and Kibana 5.3.1 with a trial license of X-Pack enabled by default, so our cluster is secured through X-Pack Security. Communication with Kibana however is not encrypted by default; for this, we can supply a certificate and private key as template parameters to be used for SSL/TLS.
Get-ACMECertificate cert1 -ExportCertificatePEM "C:\kibana.crt" -ExportKeyPEM "C:\kibana.key"
After template deployment and with a CNAME record set up against the domain for which we generated the certificate, using the domain returned in the kibana output parameter, communication to Kibana is now encrypted when accessed from our domain. A CNAME record is required here as the domain under which the public IP address is configured is owned by Microsoft Azure and hence we cannot verify ownership and generate a certificate for it. In this example, a CNAME record is set up against the sub domain arm-deployment.elastic.co with a value pointing to cloudapp.azure.com sub domain.
Too easy! Certificates issued by Let's Encrypt™ are valid for 90 days so you'd want to set up automated renewal of the certificate and key on the Kibana VM; Both the certificate and private key can be found in /etc/kibana/ssl or /opt/kibana/config/ssl for 5.x and 2.x deployments, respectively.
With the default template parameters, the Elasticsearch cluster is not directly accessible outside of the virtual network to which it is attached; only Kibana is configured with a public IP address and a network security group to allow TCP traffic on port 22 for ssh and 5601 for the web UI. External access can be configured using the loadBalancerType parameter, with a value of "gateway" deploying Application Gateway for external access, providing a means to encrypt communications and perform SSL offload, the latter feature being something that can't be done with Azure's layer 4 load balancers.
Get-ACMECertificate cert1 -ExportPkcs12 "C:\application-gateway.pfx" -CertificatePassword "CertPassword123"
Both the PFX certificate and password can be supplied as parameter arguments to the template.
Application Gateway connects to the node VMs through an internal load balancer, making it easy to add additional nodes to the load balancer backend pool to scale up the cluster.
One important point to note with Application Gateway currently is that the health check probe it uses to ascertain that the resources to which it is connected are healthy uses HTTP protocol and does not allow authentication credentials to be passed for the check. Consequently, to allow Application Gateway to work with a cluster secured with X-Pack Security, anonymous access is enabled and assigned a role with access only to the base URI. Hopefully, health check probes will support passing authentication credentials in the future, allowing all cluster resources to require authentication to access.
The vmDataDiskCount parameter can accept several values ranging from 0 to 40 to control the number of attached disks. When more than one disk is specified, a RAID 0 array device is configured to stripe data across all attached disks, to maximize IOPS throughput.
When no disks are attached, the temporary drive is configured as the location to store data, using path.data in the elasticsearch.yml configuration file. Azure mounts the temporary drive at /mnt with directory /dev/sdb1 for Linux VMs. The temporary drive is ephemeral in nature and data on it can be lost under a number of different scenarios, including when you resize the VM, shutdown or restart the VM, when the VM is moved to a different host server, amongst others. Because of this, great care should be taken if you are considering using it, to avoid data loss, such as having a sufficient number of data nodes within the cluster with replica shards. Nevertheless, using the temporary drive can provide a cost effective way of having good performing storage, since the temporary drive is provided at no extra cost and is present on the physical machine that hosts the VM.
All of the scenarios demonstrated so far have been concerned with changing the deployment options for the ARM template and deploying resources configured by the template based on those options. Where the true power of the template really comes into its own however is in the ease with which it can be combined with other Azure resources. Let's explore how we can integrate other resources with the template now.
If need be, we can create the Storage account with Zone redundant (ZRS), Geo redundant (GRS) or Read-Access Geo redundant (RA-GRS) storage for durability and high availability.
Deploying two clusters into an existing virtual network might be something you'd like to do when that virtual network already has devices attached that will communicate with the deployed clusters, such as Azure App Services. Configuring an existing virtual network for usage with the cluster requires a subnet be specified to which to attach the deployed resources and a known available private IP address to use as the static address for each internal load balancer. Of course, the subnet also needs to have a sufficient number of available IP addresses for all of the resources that will be attached to the virtual network.
Write-Error -Message "Requested $Count available addresses but subnet contains maximum $subnetCount addresses"
Write-Error "Insufficent available addresses in subnet. Requested $Count, available $($availableAddresses.Count)"
"vNetLoadBalancerIp" = "$(($availableAddresses | Select -First 1).IPAddressToString)"
Write-Host "[$(Get-Date -format 'u')] Deploying first cluster"
Write-Host "[$(Get-Date -format 'u')] Deployed first cluster"
Write-Host "[$(Get-Date -format 'u')] Deploying second cluster"
Write-Host "[$(Get-Date -format 'u')] Deployed second cluster"
The most important point here is that each cluster must be given a unique value for vmHostNamePrefix; since nodes discover each other and form a cluster using unicast hosts in elasticsearch.yml configuration, and host names are used within this configuration, the host names must be different for the two deployed clusters.
We expect a very common scenario in using Elasticsearch in Azure will be in connecting existing Azure App Services to a VNet in which a cluster is deployed, to allow those services to communicate with the cluster. This communication can be within the internal network only, meaning the cluster is not exposed externally, keeping all data transfer within Azure.
There are a few resources involved to put this altogether, so we'll go though them piece by piece. For this example, we're going to deploy a Nancy web application that provides a search interface over Nuget packages. Thankfully, we don't need to write this from scratch as we can use our Nusearch tutorial repository in conjunction with the Azure quickstart template to deploy a web application from github.
Write-Host "[$(Get-Date -format 'u')] Deploying new storage account"
Write-Host "[$(Get-Date -format 'u')] Deployed new storage account"
Write-Host "[$(Get-Date -format 'u')] Deploying cluster"
Write-Host "[$(Get-Date -format 'u')] Deployed cluster"
WriteLog "Finished Deploying Nusearch website" -ForegroundColor "green"
WriteLog "Creating a public IP address for this Virtual Network"
WriteLog "Finished creating a public IP address for this Virtual Network" -ForegroundColor "green"
WriteLog "Adding a root certificate to this Virtual Network"
WriteLog "Finished Adding a root certificate to this Virtual Network" -ForegroundColor "green"
WriteLog "Creating Virtual Network Gateway. This may take up to an hour."
WriteLog "Finished creating Virtual Network Gateway" -ForegroundColor "green"
WriteLog "Creating Virtual Network Connection for website $webAppName to Virtual Network $($vnet.Name)"
WriteLog "Virtual Network Connection created" -ForegroundColor "green"
WriteLog "GatewaySubnet already exists for Virtual Network $($vnet.Name). Using this." -ForegroundColor "green"
WriteLog "Creating GatewaySubnet in Virtual Network $($vnet.Name)"
WriteLog "GatewaySubnet created in Virtual Network $($vnet.Name)" -ForegroundColor "green"
WriteLog "Retrieving VPN Package and supplying to Web App"
WriteLog "Adding website $webAppName to Virtual Network $($vnet.Name)"
WriteLog "Finished adding website $webAppName to Virtual Network $($vnet.Name)" -ForegroundColor "green"
Setting up VNet integration and configuring point to site connectivity can take up to an hour to deploy and configure so don't worry if the process takes up to this amount of time.
WriteLog "Updating App Settings for website"
WriteLog "App Settings for website updated" -ForegroundColor "green"
After performing the network sync, the App Service can now talk to the cluster!
And we now have an Azure App Service talking to our Elasticsearch cluster on Azure, allowing us to search across Nuget packages with autocompletion.
We've walked through a few different scenarios, demonstrating the capability of our ARM template. Check out our Azure Template examples repository to play with any of the examples presented here. With a few lines of PowerShell or the Azure CLI, it's easy to integrate Elasticsearch and Kibana with your existing Azure resources, opening up new opportunities to splice, dice and analyze your data with the power of the Elastic stack.