What is GitOps?

So what actually is GitOps? On the surface, it is quite simple. GitOps is centered around using a version control system (such as Git) to house all information, documentation, and code for a Kubernetes deployment, and then use automated directors to deploy changes to the cluster. GitOps provides a way for developers to manage operational workflows, particularly for Kubernetes, using Git and their own version control system.

In short, the same process you use to merge code using pull requests is used to deploy workloads to Kubernetes.

Continuous Integration

Like any good pull request (PR) workflow we need to be running continuous integration tests to validate our changes are mergeable into our codebase. This is no different for the Kubernetes manifests we are maintaining in source control.

All our custom helm charts at Mettle are stored in a centralized repository called mettle/k8s-helm-charts. These charts are then deployed into our Kubernetes clusters using Flux HelmReleases, to read more about these please see https://docs.fluxcd.io/projects/helm-operator/en/latest/references/helmrelease-custom-resource.html

The CI problem

We validated our helm charts by executing helm template and then piping the output into kubeval (https://github.com/instrumenta/kubeval) for each chart in turn (see below) to verify the manifests align to the schema definitions for the specific version of Kubernetes.

The issue was that every time kubeval was being executed it seemed to be taking ages, we realized this was because kubeval was having to pull down all the schemas each time to verify the helm chart against. This meant if we added more charts to the repository it was only conflating the issue.

Schemas: https://github.com/instrumenta/kubernetes-json-schema

The CI solution

When looking at the flags available to Kubeval I noticed the following one:

-s, --schema-location string Base URL used to download schemas. Can also be specified with the environment variable KUBEVAL_SCHEMA_LOCATION.

If we could download the necessary schemas locally and then point kubeval at them this would drastically decrease the amount of time required to lint our helm charts.

I went about creating a container called kubernetes-toolkit which would contain all the packages and binaries we use to validate kubernetes manifests (e.g. kubectl, helm and of course kubeval).

Additionally and most importantly, I would download the schemas for the specific version of kubectl inside the container at build time. The bash script for this can be seen below

The version of Kubernetes is passed to the Dockerfile as an argument at build time to make sure the correct version of kubectl is installed and most importantly only the schema definitions for that version are stored in the container.

The logic to maintain the specific set of schema can be seen below:

Additionally, we need to tweak the bash script which runs kubeval to the following:

We now set the KUBEVAL_SCHEMA_LOCATION environment variable to make sure the schemas inside the container are used. Additionally, we specifically tell kubeval what version of Kubernetes to validate against using the

—- kubernetes-version flag, this is required as the container only contains version 1.17 schemas.

Finally, we need to hook this all up in our CircleCI configuration file (below)

We aim to build a new version of the kubernetes-toolkit container for each release of kubernetes and update our corresponding CI jobs accordingly.

Shoutouts

I would like to shoutout https://twitter.com/karlstoney for this idea it's reduced our build times from 20 minutes to around 5 seconds with minimal changes.