When one thinks of Kubernetes and deploying stuff into Kubernetes, one of the usual ways to get such stuff into Kubernetes is through the use of Kubernetes manifest files. Kubernetes manifest files describe various different resources in Kubernetes cluster - some primary examples that are often used are Deployment, Configmap, Secret, Service and even Ingress Kubernetes resources/objects.
When building an application, a common way to alter and set the running properties of the application is to use configuration files that could be written with JSON or Yaml files. This is the same even if the application is simply deployed in a Virtual Machine or even in a container within a Kubernetes Cluster. The general assumption is that the configuration file does not change that often - if the configuration files is to be change, the usual way to have the application conform to the new configuration file would be stop the currently running the application and start it once more.
In the real world, we often have to deal with such large traffic loads that it is almost necessary to know that there is possibility that we might need to get data stored in a cluster of machines. In the case if we have applications that barely need to deal and manage data, we can simply on existing products out there that can simply scale out the number of replicas of the application which it can simply serve pretty easily. However, what about applications that rely on database? We need our database server cluster to also scale out accordingly as well (there are limits to scale vertically in most cloud providers after all)
This is more of a reminder post for me that every aspect of application development is critical and sufficient thought should be put behind it. This time around, it’s on database migration within applications.
When building login systems in applications, there are generally two parts to it; authentication and authorization. Authentication is the step to provide and identify who the user that is attempting to use the system. Authorization is the step to decide whether user that is using the system is “allowed” to access or modify a particular resource on a system.
I was watching a bunch of tiktok and youtube videos recently and kind of started to wonder how such companies serve videos to their consumers. That is where I started to going down the rabbit hole of how videos are served and how to try to ensure the possibility that videos can be played without requiring to download the entire video.
For many people, cooking is not just a means of sustenance but a beloved hobby and a way to express creativity in the kitchen. However, one of the biggest challenges for home cooks is keeping track of their recipes and possibly the list of interesting recipes from other people. In my opinion, it’s general a good idea to have a copy of such information on hand (since websites/videos hosting such recipes can eventually disappear). However, recording such information in plain text might be a tad “boring” - it’s also harder to kind of parse as well as process further. In this blog post, we will explore using cooklang as a possible tool to “standardize” such information.
Introduction # I used to work with Google Analytics to obtain site analytics for websites and android application. Technically, the current blog is monitored using Google Analytics. Monitoring of website data is generally useful as it provides information to the authors of the website/website owners on what particular content that website visitors find the most useful. With such information, it makes easier for the owner to try to add new content that attempts to provide such relevant content to visitors which would hopefully spur a virtuous cycle of gaining more audience for the website.
Over the recent weekends, I’ve decided to take a gander and try another “serverless” tool called Google Cloud Workflows. The tool’s appeal is to be able coordinate a bunch of services in order to achieve a particular goal. The coordination effort (or workflow) can easily get pretty complex -> one way would be to script but if we want to have the capability to have the button to run the entire workflow from start to end with logging in place as well as capability to run the workflow based on particular triggers.
The leader election mechanism is a somewhat complex thing to kind of code up for an application. There are various Golang libraries that assist with this but it would be nicer if there were mechanisms within the environment that the application operate in which can help with this. In the case for the Kubernetes ecosystem - we can actual rely on the fact of how Kubernetes would usually etcd that does this leader election dance on our behalf. If we can tap on this mechanism, we can avoid introducing this mess of a complexity within our application.
The whole process of profiling an application is an attempt to identify hotspots within the application which consumes more resources or takes too much time - knowing this would allow us to identify how to further improve the code within the applications that we build in order to build applications that consume less resources or would respond better to external inputs. Profiling of an application is just another aspect to improve observability of application’s performance on top of the common usual tooling such as distributed traces, metrics and logs. Tools such as distributed traces, metrics and logs only can capture part of the picture of how an application performs within an environment but is different for profiling. Profiling would point out what is happening “internally” within the application such as amount of memory being allocated for particular functions, how much CPU time is being taken for a particular function, thereby providing even more visiblity to how the application works.
A friend of mine once mentioned about one of the tasks that he had to go through during his programming days was to build out a server which would respond to the redis-cli tool and I started to think - “that’s something I’ve never done before… I wonder how hard it is?” After a day of tinkering around - it’s definitely something that’s not “intuitive” to immediately get done; there are definitely some concepts that I’m not super clear about but it’s definitely something that can be slowly built out while learning various concepts.
Over the past month, I decided to go down the rabbit hole of exploring an example of a self balancing tree data structure. I generally don’t need to handle data structures on a day to day basis - I mostly deal with integration of tools as well as deployment of tools into a Kubernetes cluster. However, even if I don’t deal with that side of things, I do find that some of the thought process behind the data structures and algorithms are pretty interesting. (I’m still kind of waiting for a moment where I can actually utilize it in my work for real in a way)
There is a trend of images that follow the philosophy of minimizing the size of image by removing almost everything out of image. This helps with getting image downloaded more quickly by kubelet into the nodes as well as possibly reducing the attack surface of the container even further (I suppose it’s harder to do things in a container if utilities like shell or bash don’t exist within it). You would probably see errors such as this for those containers that have somewhat remove the shell/bash:
While dealing with branded links during my course of work, I kind of wondered how it can be tackled if I were to do it in a Google Kubernetes Engine Cluster. The situation I would imagine that would need to solve is this:
This is a quick sample tool to retrieve bus arrivals in Singapore. In order to use it, we would need to find for the Bus Stop ID or Bus Stop Code from where we’re taking the bus from. After keying it, it would fetch the records from LTA Datamall’s real time bus arrival API and present those records in this tool.
Database migration is kind of a critical bit when it comes to running and operating applications. In Golang, it is kind of appealing to rely on ORM (Object Relational Mapping) libraries. It allows one to kind of map structs to tabular structures within the database storage. One such example of an ORM library that I’ve found on the first page of Google is GORM.
I am still building up my personal pet project: https://github.com/hairizuanbinnoorazman/slides-to-video; the aim of this project is a personal one - to build up a set of microservices that is able to be deployed in various ways such as locally via Docker Compose or even to Kubernetes or the serverless Cloud Run platform on Google Cloud Platform. There was a previous blog post describing an initial part of this journey: Lessons on building the project - Part 1
While building Elm based frontends, I decided to take the opportunity to learn on how to craft a chat application. Truthfully, I’ve never really built one before (nor do I need to). But it does seem like an interesting programming exercise to kind of go thru - in order to understand how such applications are built, deployed, scaled and managed. For the frontend, I’m mostly set to use Elm (probably you’ve seen a previous post on my “dislike” for other Javascript based frameworks, which is essentially all the popular ones in the market). For backend, I will probably stick to Golang since that is the language I’m most comfortable with (all hail statically typed languages)
What and why systemd? # Systemd is a convenient set of tooling that can be used to manage services and applications on a linux server. When we are managing applications on a server, we would want the following properties automatically for most application - the requirements are somewhat for most applications: