Kubernetes on Experiment, Fail, Learn, Repeat

Deploying Agent Sandbox with gVisor and Memory Snapshots on GKE Standard

Wed, 24 Jun 2026 00:00:00 +0000

Agent Sandbox provides a Kubernetes API for stateful, singleton, Pod-backed workloads. GKE can add a stronger gVisor isolation boundary and Pod Snapshots that preserve process memory and root filesystem changes.

GKE as Gitlab Runner

Tue, 08 Apr 2025 00:00:00 +0000

Part of my job involves me dealing with Gitlab on a daily basis. Gitlab is a complicated beast to handle and it took a while to get around the various features that the product offers. One of the offerings available is one where we can set an entire Kubernetes cluster as a potential target where we can then create containers and run tests on said cluster.

Building a code assessment tool but in Kubernetes

Sun, 10 Nov 2024 00:00:00 +0000

I had previously attempted to build a code assessment tool in docker. That involves doing the following:

Access Cloud SQL from Google Kubernetes Cluster without Cloud SQL Proxy

Wed, 20 Sep 2023 00:00:00 +0000

Introduction
#

Similar to my previous blog post, we would usually be connecting Google Kubernetes Engine (GKE) clusters to Cloud SQL databases by using the Cloud SQL Proxy. However, we can now use Private Service Connect, which allows for private communication between different Google Cloud services, similar to how we did for connecting our application in Google Compute Engine (VM) to a Cloud SQL instance.

Storing Helm in Docker Registries

Wed, 23 Aug 2023 00:00:00 +0000

We can apparently now store helm charts in Docker registries - this was made available via helm commands since v3.8.0. https://helm.sh/docs/topics/registries/

Now with that being available for use, we can now use it across a variety of storage mechanism (as compared in the past when the artifacts produced through it has to be managed in some of file system and would require some of index file to list all available helm charts available).

Deploy Golang Apps that interact with headless service in Kubernetes

Wed, 12 Jul 2023 00:00:00 +0000

In certain application scenarios - there is a need to have applications that need to do client side load balancing to a bunch of servers. Such cases are pretty rare - but we won’t be covering the exect reasons or scenarios or when these are needed. Instead, we will cover how we can do so with Golang applications in Kubernetes cluster.

Running database migrations in Helm chart

Wed, 07 Jun 2023 00:00:00 +0000

In many examples for helm charts, the general focus is on the “2nd” day operations of having applications running without too much issues. In the case for usual web developers, that would mean applications handled with kubernetes deployment objects which would run a set number of replicas (or handled via HPA) in the kubernetes cluster.

Altering outputs of helm installations with post-renderer via kustomize

Wed, 24 May 2023 00:00:00 +0000

When one thinks of Kubernetes and deploying stuff into Kubernetes, one of the usual ways to get such stuff into Kubernetes is through the use of Kubernetes manifest files. Kubernetes manifest files describe various different resources in Kubernetes cluster - some primary examples that are often used are Deployment, Configmap, Secret, Service and even Ingress Kubernetes resources/objects.

Updating configuration in Kubernetes pods managed via Helm

Wed, 17 May 2023 00:00:00 +0000

When building an application, a common way to alter and set the running properties of the application is to use configuration files that could be written with JSON or Yaml files. This is the same even if the application is simply deployed in a Virtual Machine or even in a container within a Kubernetes Cluster. The general assumption is that the configuration file does not change that often - if the configuration files is to be change, the usual way to have the application conform to the new configuration file would be stop the currently running the application and start it once more.

Running kubectl in a Kubernetes Pod

Wed, 12 Apr 2023 00:00:00 +0000

I have a little side project at work where it somewhat requires me to allow a pod within a kubernetes cluster to access and query and manipulate resources in a Kubernetes cluster. This would provide some sort of special development environment within pod with the required capability to update the cluster. In order to do this, we need to add a bunch of roles, clusterroles and its bindings (essentially the RBAC system in Kubernetes) to allow the pod to access said resources

Leader Election in Kubernetes via Kubernetes Configmaps and Leases

Sun, 28 Aug 2022 00:00:00 +0000

The leader election mechanism is a somewhat complex thing to kind of code up for an application. There are various Golang libraries that assist with this but it would be nicer if there were mechanisms within the environment that the application operate in which can help with this. In the case for the Kubernetes ecosystem - we can actual rely on the fact of how Kubernetes would usually etcd that does this leader election dance on our behalf. If we can tap on this mechanism, we can avoid introducing this mess of a complexity within our application.

Continuous Profiling of Applications in Kubernetes via Pyroscope

Fri, 05 Aug 2022 00:00:00 +0000

The whole process of profiling an application is an attempt to identify hotspots within the application which consumes more resources or takes too much time - knowing this would allow us to identify how to further improve the code within the applications that we build in order to build applications that consume less resources or would respond better to external inputs. Profiling of an application is just another aspect to improve observability of application’s performance on top of the common usual tooling such as distributed traces, metrics and logs. Tools such as distributed traces, metrics and logs only can capture part of the picture of how an application performs within an environment but is different for profiling. Profiling would point out what is happening “internally” within the application such as amount of memory being allocated for particular functions, how much CPU time is being taken for a particular function, thereby providing even more visiblity to how the application works.

Restricting connections between pods in a Kubernetes Cluster (Network Policy)

Sun, 15 May 2022 00:00:00 +0000

There is an old adage from security land that we should restrict access to resources/assets as much as we can. Users and applications should only access items that they need to operate themselves. Following this line of thought, that would mean that if we are to deploy application in a Kubernetes Cluster, we should ensure that pods should only accept communication that they’ve explicitly declared as “required”. Is there a way to do so?

Debugging Distroless Kubernetes Pods

Fri, 15 Apr 2022 00:00:00 +0000

There is a trend of images that follow the philosophy of minimizing the size of image by removing almost everything out of image. This helps with getting image downloaded more quickly by kubelet into the nodes as well as possibly reducing the attack surface of the container even further (I suppose it’s harder to do things in a container if utilities like shell or bash don’t exist within it). You would probably see errors such as this for those containers that have somewhat remove the shell/bash:

Devops Interview Questions

Sun, 27 Feb 2022 00:00:00 +0000

This is a list of notes for possible interview questions with regards to devops roles. Interview questions for devops are particularly hard to cover since devops roles generally cover a broad range of topics and technologies. I will update this page as I see any interesting or “hard” questions to cover.

Kubernetes Ingress for applications with branded links on GKE

Sun, 13 Feb 2022 00:00:00 +0000

While dealing with branded links during my course of work, I kind of wondered how it can be tackled if I were to do it in a Google Kubernetes Engine Cluster. The situation I would imagine that would need to solve is this:

App with Metrics, Logs and Distributed Traces

Wed, 29 Sep 2021 00:00:00 +0000

In a previous post, it details some information of how to setup some open source tooling to capture logs, retrieve metrics as well as capture distributed trace information from apps. The previous blog post would cover the setup of logging system which is Loki, distributed tracing system which is Tempo and metrics collection system which is Prometheus. Refer to the link below here.

Setting up Observability Tooling in GKE

Wed, 01 Sep 2021 00:00:00 +0000

Generally, most cloud providers come along with all the observability tooling that you need for your apps built-in with the platform. Some of the common observability tools such as logging, monitoring and nowadays, distributed tracing are usually made available and you can easily use said tools by reading up on the various documentation of how to setup each of these tooling. E.g. if your application is inside a virtual machine and if you need collect metrics and logs from the application, you may need to install an agent in the said VM. The agent would collect those information and send it to the centralized observability tooling in the cloud provider where the information would be provided to you via a UI. Most of the time, these tools are charged based on the amount of logs/metrics you generate from the application (so the less logs/metrics you generate, the cheaper it is monitor your application - a very understanable/reasonable situation). In cases where if your application runs in Kubernetes, maybe the cluster comes with agents pre-installed, making it easier to make use of the logging/metrics/distributed tracing that the cloud provider has.

Using Envoy for GRPC Applications in Kubernetes

Thu, 15 Jul 2021 00:00:00 +0000

As of now, one of the common and easier way to have services communicate with each other would be over HTTP. In real world use cases, HTTPS is usually used (in order to ensure communications are secure) and this communication is done following some sort of REST framework. This provides some sort of structure of how to standardize such communications for the various software applications out there. It got to the point where entire companies are developing in order to support this: e.g. Apigee, SmartBear

Python Flask Apps in Kubernetes

Sun, 18 Apr 2021 00:00:00 +0000

This are some notes in the case where one wants to deploy a bunch of python “microservices” to a Google Kubernetes Engine cluster. These notes emphasize on the basics rather than the various nuances of running a “production” grade python application.

Tripping over helm hooks

Fri, 11 Sep 2020 00:00:00 +0000

One typical way to do packaging for applications that are to targeted to be deployed into a Kubernetes environment would be to utilize the helm tool. The helm tool has been used widely enough to the point that there are whole ecosystems that support the usage of this tool. Refer to the website for this here: https://hub.helm.sh/

Attempting to understand CSI Kubernetes

Fri, 01 May 2020 00:00:00 +0000

NOTE: THIS POST IS NOTES IM TAKING FOR MYSELF WHILE ON THIS JOURNEY. TAKE IT WITH A BAG OF SALT. NOT ALL THINGS MENTIONED HERE IS TRUE - DO YOUR OWN DUE DILIGENCE

Trying Distributed Tracing with Jaeger

Sat, 04 Apr 2020 00:00:00 +0000

Let’s say we have a set of applications that was designed to be a set of microservices. Each of the applications would generally be designed to be focused on one specific domain and in order to achieve the overall goal of the platform. However,for the platform to work properly, the applications would generally need to work together as one which would involve the application contacting each other.

Attempting to setup Kubernetes on Ubuntu VMs

Wed, 05 Sep 2018 00:00:00 +0000

This post details my naive attempt to bring up a Kubernetes cluster on a VM. These steps to try out Kubernetes in a bare Google Virtual Machine (but the following steps should work for most Debian/Ubuntu virtual machines). This deploys a single node Kubernetes cluster (naturally don’t think of using this for production)

Trying out skaffold

Thu, 09 Aug 2018 00:00:00 +0000

When developing application that are meant to be deployed to the Kubernetes platform, it involves a bunch of steps on top of your usual local development work:

Writing a Dockerfile to package the application (Multi stage applications are optional here - useful for compiled based languages)
Build and tagging the docker image of the application with the target repository
Either use kubectl commands or use kubernetes config resource files to define the resources required for deploying the applications. Use those commands/configurations to define the resources on the staging/production application
Repeat the process for each update of the application (Repeat second point onwards)

As you see from above, it starts to be pain to do so after each iteration of the application development. The building of the docker containers process as well as the applying of the new images to each cluster, (sometimes with slightly changed configuration files) - the kubernetes secret and config files can change across different environments.

Lessons from Kubecon/CloudNativeCon 2018 Europe

Wed, 16 May 2018 00:00:00 +0000

The following set of summaries are from the Kubecon and Cloud Native Con Europe in Denmark from 2-4 May 2018.

These summaries are from conference talks that I thought provided more interesting thinking points.

Summaries from Fossasia conference in Singapore

Wed, 02 May 2018 00:00:00 +0000

View the full list of Fossasia Video recordings on the [engineers.sg] website on this website: https://engineers.sg/conference/fossasia-2018

Here are some of the videos I particularly liked. I summarized some of the interesting points from said websites.