Ask HN: Which VM orchestrator do you use?
&quot;It depends&quot;... as usual.<p>Here comes a long one, but tl;dr: make sure you know your goal before you select your tools.<p>Orchestration is a somewhat broad concept and can be interpreted at many levels. From a management perspective it is about what processes are acting to the benefit of the organisation to fulfil a business need, regardless of the amount of automation. Having a room full of humans vs. a couple of binaries and event logic doesn&#x27;t matter if the orchestrated result is the same.<p>Now, when it comes to your extra information you seem to be using microservices, but deploying them as individual virtual machines instead of just containers. While I applaud the choice of better isolation, virtual machines isn&#x27;t the only way to do this. Firecracker does this too but much more efficient with the same protections (<a href="https:&#x2F;&#x2F;firecracker-microvm.github.io" rel="nofollow">https:&#x2F;&#x2F;firecracker-microvm.github.io</a>). So before going all-in on virtual machines, think about your goal and check to see if this is really the best way to do it. General business applications really don&#x27;t have this level of isolation as the main factor in safeguarding applications as there are many other more fragile layers where the actual problems occur. A well-isolated application with no AAA on its interfaces might as well not be isolated at all from a security perspective.<p>The selection of tools is probably going to be influenced by what else you are going to do. If you have automated infrastructure you&#x27;ll probably want to tie in to that. Some automations have to trigger at deploy-time to (de-)provision at the right moment, while others have to simply react to the fact that something is needed and autonomously scale with the requirements.<p>If your (micro)service needs a database, it probably also needs configuration to access it, and the database needs to be configured as well. Perhaps there are some networking components, and storage if you have object or block persistence needs.<p>If you are running on AWS for example, your &#x27;orchestrator&#x27; is EC2, but you still need to &#x27;orchestrate the orchestrator&#x27; to start the correct configuration with all its dependencies prepared ahead of time. You&#x27;ll probably want to prepare the virtual machine image so it actually contains everything it needs during build-time so it can just do the running at run-time. Packer comes to mind for such a job.<p>Most systems that are &#x27;good&#x27; got there because they had a lot of usage, because that is what generates data (feedback) on how well it does the job it needs to do. With that larger amount of usage comes a higher load from stakeholders and with that there tends to be a growth of commercial incentive turning a lot of products into some collection of commercial side-options or main-options.<p>To get a bit closer to a possible answer, I&#x27;ll sketch out two scenarios, a managed one and an unmanaged one (from a Layer 1 and 2 perspective, hardware).<p>Scenario 1: Mostly managed primitives.<p>Depending on what you already have you are likely to have an API that interfaces with the storage, networking and compute resources that are available to you. Depending on the systems, Terraform and Ansible can configure and prepare almost anything to your liking and make sure it stays in the desired state.<p>When deploying an application as a virtual machine image, you&#x27;ll probably have the following events happen:<p><pre><code>  1. SCM change triggers application build (Git and some CI solution)
  2. Successful build triggers next integration step (CI)
  3. Integration takes your build output and wraps it in a virtual machine image (Packer)
  4. Successful integration triggers deployment step (CI)
  5. Deployment step contacts your API of choice to let it know about the new image (Salt&#x2F;Ansible&#x2F;Terraform&#x2F;FaaS&#x2F;Webhook)
  6. API registers the new image and you can start rolling over instances until all the old ones are gone

</code></pre>
Scenario 2: You have to DIY everything.<p>Steps in there are essentially things you have to do before you can start thinking about scenario 1 which is what follows after this.<p><pre><code>  1. Make your hardware boot from the network
  2. Setup a network boot server
  3. Prepare self-starting hypervisor images of your choice (KVM-based, Xen-based, Hyper-V-Base, vmWare-based)
  4. Make sure your DHCP and&#x2F;or TFTP servers have logs you can access&#x2F;parse
  5. Once a server boots, it loads a self-starting image and depending on how you configured the images they will start out &#x27;empty&#x27; with a well-known key or other credential
  6. Use something like Ansible or SaltStack to discover any servers and select the &#x27;empty&#x27;&#x2F;&#x27;fresh&#x27; ones
  7. Depending on your selected hypervisor, you&#x27;ll now have to connect to a management API, control plane or simply write out a list of nodes somewhere (useful for small-scale SaltStack setups!)
</code></pre>
At this point, the &#x27;compute&#x27; is done and ready for deployments. A big issue here is that we have many assumptions and the amount of new variables are so large that it&#x27;s hard to get some more steps in.<p>Next, you&#x27;ll need storage which is much too broad to cover here, but it generally ends up with some sort of redundant NFS, iSCSI or more modern setup like Ceph. Those are all configurable via SaltStack or Ansible.<p>You&#x27;ll want networking after that, something like a vswitch with VLANs maybe. A very simple way to do this is start an OpenWRT or OPNSense VM with a bunch of pre-loaded virtual interfaces, subnets, and firewall rules. How this exactly works depends very much on what hypervisor and control plane you have. VMWare with NSX is very different from Proxmox with openvswitch (but they do the same thing).<p>When you now want to &#x27;deploy&#x27; a VM, you use your Ansible&#x2F;Salt&#x2F;Terraform-enabled infrastructure where you can &#x27;update&#x27; the base configuration of a VM-based application and let Ansible&#x2F;Salt&#x2F;Terraform create new VMs and destroy the old ones. Configurations might be flat files written out to the network-booted compute nodes, or some configuration database in a controlplane behind an API.<p>Now, all of this can be partially homelab-automated with things like Proxmox, Xen XCP-ng, VMWare etc. But as soon as you actually need more facilities like actual API-configued load balancers, databases, metrics, discovery you&#x27;ll start home-cooking a complete OpenStack replacement. At that point, just get OpenStack or realise that the benefit of scale you get with Azure&#x2F;GCP&#x2F;AWS lets you do what makes you money instead of doing busywork that doesn&#x27;t actually matter in the end in many cases.