Every platform has limits; workstations and physical servers have resource boundaries, APIs may be rate-limited, and even the perceived endlessness of the virtual public cloud enforce limitations that protect the platform from overuse or misuse (Azure, AWS, GCP).When working on scenarios that take platforms to their extreme, those limits become real, and therefore thought has to be put in to overcoming them.
The following post, which includes essential notes taken from my work with Mike Kiernan, Mayur Dhondekar and Idan Shahar, covers some iterations where we try to reach a limit of 10K virtual machines running on Azure and explores the pros/cons of the different implementations.
Load and stress tests before moving a new version to production are critical on the one hand but pose a real challenge for IT on the other, because they require a considerable amount of resources to be available for only a short amount of time, every release-cycle. When purchased, the infrastructure doesn’t justify its cost over extended periods, making this a perfect use-case for a public cloud platform where payment is billed only per usage.
DDoS simulation – note that in this case ethics must be practiced and the targeted endpoint should be owned by you, otherwise you risk the liability for damages.
Trigger the load-test process using HTTP POST.
Avoid manual steps, pre-requisites and custom images which may be outdated over time.
Minimal total cost of the solution, which includes the used infrastructure resources as well as additional management or monitoring services.
Flexibility and Cost – Use any type of VM, including B series VMs (Burstable type), Reserved Instances or Low Priority, and maintain control over additional resources such as public IPs.
Monitoring – Using Azure Monitor Log Analytics to keep track of running VMs in the cluster.
Scale – Scale out and reach a full sized cluster faster than any other platform we’ve tested during the process.
There is no elasticity in the cluster once provisioned making Autoscale features of VMSS are useless.
VMSS limits to 1,000 VMs per set and Azure VNet limits to 4,096 private IPs will force managing multiple resources and require further automation to provision and de-provision.
Keeping the solution minimal in code, using only an ARM template containing a single VM and a PowerShell that iteratively deploys it.
However, given Azure Resource Manager (ARM) API’s 1200 write-requests per hour rate limitation, we can quickly figure out that deploying a single VM, which actually requires five resources written (VM, Disk, NIC, PIP, VM extension), would be rate-limited to about 240 VMs per hour, whichever way we’ll try it; the portal, CLI, PowerShell, ARM templates (with or without copy-indexes), etc.
Scaling above 4,096 VMs requires orchestration of more than one VNet.
Avoid using custom images and other manual prerequisites to allow easy integration with ci/cd pipeline tools.
Avoid ARM API throttling when provisioning large scale-sets by pausing for 5 minutes between actions.
Each scale set holds up to 600 VMs. This number is configurable but is a recommended number to use for best results using a marketplace image. Note that for a custom image that number should be configured to be smaller.
Downloading application and data content from a single storage account quickly proves unscalable as application packages grow and Azures blob storage read limits start kicking in.
The client application generates a load on the backend servers (5).
First scale up from 100 instances to 2K in 55 minutes.
Second scale up to 10K VMs completes within 2:45 hours.
De-provisioning the entire cluster took less than 1:30 hours.
Azure Monitor Log Analytics, in this case, is much cheaper, though providing much less visibility to the health of each node as we only listen for heartbeat messages.
Provisioning time is minimal and most cost is billed when the cluster is already at full scale.
Using burst type (B-Series) VMs to achieve the lowest rate.
This section describes the design considerations for the code that operates this environment which is hosted on an Azure Function and uses the new Durable Functions Framework.
Durable functions are a nice addition to Microsoft’s serverless framework, introducing state and orchestration to the somewhat limited space of “serverless”. I think they allow serverless to become more than just “integration code” that the pattern sometimes falls into, and can even in some cases be used to implement a full microservices-based architecture and orchestration.
Durable functions define a system in which three function-sub-types play different roles in the orchestration process.
Activity Function – The basic unit of work. Usually, a solution is built using some Activity Functions, each does one “thing”, and they are orchestrated in some way.
Orchestrator Functions – The orchestration logic of the process. Orchestrator Functions can orchestrate Activity Functions or other sub-Orchestrator Functions; they can also trigger “wait” on events or timers in which the function is not active and not billed, using a simple async-await clause. When running, they are managed and interfaced by an orchestrator client component with a unique ID.
Client Functions – External interface to a durable-function such as HTTP request, queue, etc. that trigger an orchestrator function or monitor running orchestrator client.
By using these types, new patterns emerge that enrich Functions greatly. Specifically, we’ll be using the fan-out/fan-in pattern described here, but instead of implementing a parallel fan-out, we’ll run an orchestrated and timed sequenced fan.In addition, the orchestration process is built (behind the scenes) using storage queue which provide some level of inherent error handling in case of action failures.
You can find the code and deployment scripts that were written to enable this solution (codenamed “buzz”) on GitHub. The repo’s readme file contains detailed instructions for deploying the solution to your account, and though the implementation uses Windows 10 and Windows wget client application to generate load on the input endpoint, it is quite easy to change that to any client application or logic required by changing the PowerShell scripts which execute on each node.
ARM Template – provisioning all required resources for a single VMSS.
Azure Function – A durable, timed ARM template deployments orchestration.
Partition the cluster based on the configured VMSS size.
Iterate, Orchestrate and wait between the different steps to provision or de-provision each VMSS and its related data.
I found myself challenged more than once while implementing this project in a way which takes into account the limitations and thresholds applied near the edge of the cloud platform’s capabilities, but also with optimization of cost, performance and usability. The end solution uses serverless hosting to automate and orchestrate the provisioning process, and provides excellent results in terms of scale, agility and cost.
Azure Durable Functions provides robust yet lightweight orchestration for the complicated provision process, with very little plumbing required.
Design and technological decisions are adapted to requirements and limits, this is natural and implementation agility is crucial.
Automation, process and code are key in DevOps practices. The solution always had the hidden goal of being easy to integrate into any ci/cd pipeline by using webhooks and no custom images which may refer to some manual process to create. This in itself is a value proposition of the final solution.
Could you do a follow up on the performance you got from this setup. Im particularly interested in Request per seconds that you can hit the endpoint with.