My Blog on Kubernetes
In my job as Site Reliability Engineer I deploy new or updated services with zero downtime multiple times per day. In this article I’d like to explain how I usually perform this task by using my website as example service.
The idea
for applied over-engineering to put my website on kubernetes came from this tweet
by
@dexhorthy. As you can see in the picture he tweeted, running a small website on a planet-scale orchestration platform is like driving around a small load on a flatbed truck. However, using a service with limited complexity as an example allows us to concentrate on the important aspects of this article: The build pipeline and kubernetes. Kubernetes is an open source container orchestration software which was inspired by Google’s famous job scheduler Borg.
Overview
So here is what we are going to do:
- Containerize the website using a multi-stage
Dockerfile.
- Setup automatic builds via Google Container Builder.
- Fire up a cluster on Google Kubernetes Engine.
- Configure kubernetes objects such as namespace, deployment, and service.
Here is a visualization to sprinkle some color into this topic:
Containerizing
I previously described how my website’s source is written in Markdown and compiled into static HTML using Hugo. Many of my articles in this blog contain syntax-highlighted code examples. The syntax highlighting is not performed by Hugo itself but it is done using an external library called Pygments. Rendering my website’s static HTML therefore requires both softwares to be installed, Hugo and the Pygments library. Although essential for rendering, neither of both is need for serving the website. They are considered build tools in this context, comparable to a compiler for a program. Once a deployable artifact has been created, the build tools are no longer needed and should not be deployed or be part of the service in production.
To containerize the website we need two stages:
- The first stage is a defined build environment containing all required build tools and the source of the website.
- The second stage is the build artifact (HTML and assets) and a webserver to serve the artifact over HTTP.
Probably the most famous container building solution is Docker. Starting from version 17.05 Docker supports multi-stage builds. multi-stage builds allow us to have a fully fledged build environment and still produce a lean artifact by moving specific files from one stage to the next and throwing away the rest of the stage.
Here is the multi-stage
Dockerfile I used to containerize my website:
FROM ubuntu:latest as STAGEONE # install hugo ENV HUGO_VERSION=0.26 COPY --from=STAGEONE /public/ /usr/share/nginx/html/ EXPOSE 80
In stage one we fetch the latest Ubuntu Linux and name the stage
STAGEONE.
Then we install a newer version of Hugo and fetch Pygments via the distribution’s repositories.
Unfortunately, the Hugo version that is in the repositories does not support a feature that my website needs, otherwise we would have installed Hugo via
apt-get, too.
Once the software is in place, we build the website by running
hugo.
Here the resulting folder
/public is the build artifact.
The second stage of the build is described in the last three lines.
It starts with a minimal image containing the nginx webserver and adds the build artifact from the previous stage using the
--from=STAGEONE flag.
Everything else from stage one is thrown away.
The resulting container is solely based on the last stage’s commands.
The
Dockerfile is stored in the root directory of the private
git repository that contains the website’s source code.
Build Pipeline
When a change gets merged into the master branch it is considered production ready. For a simple service like my website a change is usually a fixed typo or a new blog article resulting in a new version of the website. For more complex services I tend to build an image for every change in every branch and automatically push the resulting image through regression tests and a smoke test. But let’s not over-engineer an already over-engineered example. 🤓 So for every commit in the master branch we want a new image to be built.
Google Container Builder has become my favorite tool for building images. I especially like that I do not have to run a fully fledged continuous integration framework including build slaves and related maintenance work. All we need to create a new image is a build trigger and a source repository that is accessible from Google Cloud Platform (GCP). This can either be a repository hosted directly on the platform or at a third party supporting oAuth.
Fortunately, GitHub and GCP work well together. Here is the config for our build trigger:
We can manually trigger the build or just commit to the master branch.
Once the build was successful, we can pull the image from kubernetes (or any other container orchestration) via
gcr.io/danrl-com/website:master. The tag
master indicates the latest build of that branch.
Also very popular is the use of a tag named
latest to indicate the latest build, which may or may not be stable.
We can also pull specific versions, e.g. for performing a roll-back, using the corresponding tags instead of
master.
If we wanted to pin the deployment to the last version that was created on Nov 1 2017, we could use this URL:
gcr.io/danrl-com/website:2409248a51dd (see tags in screenshot).
Kubernetes Cluster
Now that we have our service nicely packed in an image, it is time to fire up the underlying infrastructure: A kubernetes (k8s) cluster.
For that we head over to Google Kubernetes Engine (GKE) and either use an existing cluster of create a new one. Here is my cluster configuration for reference:
My cluster is backed by one node pool of three nodes. It is possible to run smaller node pools. If you want to make use of automatic node updates without downtime, it is advised to use at least three nodes in a node pool. One node pool is usually enough, though.
Note: Be aware that while kubernetes itself is pretty cheap on GCP (even free for small clusters), the node pools may be surprisingly costly. For every node the standard Google Compute Engine pricing applies. The cluster used in this article, for example, costs me between $40 and $50 per month.
Cloud SDK and Cloud Shell
For the following parts of the article make sure you have installed the Google Cloud SDK including authorization of your Google account. If you don’t want to go through the hassle of installing the Cloud SDK you can also fallback to the Cloud Shell, a in-browser command line connected to a virtual machine with Cloud SDK already installed.
However, you would need to create the YAML files on the ephemeral Cloud Shell machine instead of your computer. Don’t forget to back them up, though, as the Cloud Shell machine will disappear for good after some idle time!
Deployment
I like to store my kubernetes files in a
k8s subfolder in the repository of the related service.
This puts them under version control and keeps them close to the service they belong to.
For multi-container applications I tend to create an exclusive repository only for kubernetes files, often distinguishing between production and test namespaces.
For our simple and small website service it is sufficient to keep the files in a subfolder of the root directory.
In kubernetes we use namespacing to separate workloads from each other. Our first action is therefore creating a namespace for the website service by defining it in
k8s/ns.yaml:
apiVersion: v1 kind: Namespace metadata: name: website
To create the namespace we use
kubectl from the command line.
$ kubectl create -f k8s/ns.yaml
We can always list our name namespaces via:
$ kubectl get ns NAME STATUS AGE default Active 19d kube-public Active 19d kube-system Active 19d website Active 1m
Let’s now deploy our application into the new namespace. For that we need another YAML file. This time we will create a so-called deployment. A deployment consists of one or more containers which make up an application. That application is then started with a replication factor. For our service we will use a replication factor of three. That is, we will run three instances of the website container. This allows us to do rolling updates to the application as well as to the underlying nodes. Given that the containers are spread across the nodes in the node pool, we can safely pull out a node of rotation for system upgrades without harming the availability of our application. We can also replace the containers with new images one by one if we want to update the application. This is slightly simplified, there is much more that affects availability here. I highly recommend reading the kubernetes documentation if you like to learn more about deployments.
Moving on, here is the
k8s/deployment.yaml file for the website application.
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: website namespace: website spec: replicas: 3 template: metadata: labels: app: website spec: containers: - name: website image: gcr.io/danrl-com/website:master imagePullPolicy: Always ports: - containerPort: 80
After applying the configuration the application will be available internally to kubernetes on port 80.
$ kubectl create -f k8s/deployment.yaml
Let’s check the result:
$ kubectl -n website get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE website 3 3 3 3 5m
The workload should now look similar to this on GCP:
Now that we have an application up and running it is time to turn it into a service. For a website that is making it available to the general public. There are different ways to publish an application in kubernetes. In our example we will use a service for that.
The corresponding YAML file looks like this:
kind: Service apiVersion: v1 metadata: name: website namespace: website spec: selector: app: website ports: - protocol: TCP port: 80 targetPort: 80 type: LoadBalancer
You may wonder: How does the service know which application it is supposed to serve?
After all, there could be multiple applications running on port 80.
This is what the selector is for. Compare the selector
app: website from
service.yaml with the label
app: website from
deployment.yaml.
This is where the connection is being made.
Let’s create the service:
$ kubectl create -f k8s/service.yml
This time we have to be a bit patient with kubernetes. It takes a while for kubernetes to allocate an external IPv4 address from GCP. After a minute, though, the services should be ready and look similar to this:
$ kubectl -n website get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE website LoadBalancer 10.11.246.107 35.185.83.13 80:32005/TCP 45s
The website is now available under the given IPv4 address over HTTP. Hooray! 🎉
Conclusion
Kubernetes is a state-of-the-art way of running services and orchestrating containerized applications. Running a small, static website via Kubernetes is a nice example project and total overkill at the same time. 🤩🤪 In one of the next articles I will show how we can bring the website service into the IPv6 world using a cloud load balancer and an ingress object. We will also add TLS certificates and automate their renewal.