I have a basic shopping list application that is available in the following code base: https://github.com/hairizuanbinnoorazman/Go_Programming/tree/master/Apps/shopping-list. This is a simple Golang application that also embeds a generated javascripts that has been transpiled into Javascript files. We can then embed the required CSS, Javascript and HTML files that would be the frontend of the shopping list. The frontend would then call some backend apis that would simply store shopping list items into some form of datastore - which in this case, is Google Cloud Datastore (a NoSQL database)
Serverless computing, as seen in platforms like Cloud Run or AWS Lambda, allows developers to run code without managing the underlying infrastructure. This is achieved by automatically scaling the resources based on the incoming requests, and users are billed based on the actual execution time and resources consumed during each function or container invocation.
Google Cloud Run is a serverless compute platform that automatically scales applications in response to traffic. It is designed to run stateless containers, meaning that the instances of your application are ephemeral and can be spun up or down as needed. This design choice has implications for data storage, particularly when it comes to persistence.
The typical way to access Google compute instances from Cloud Run is usually done via the Serverless VPC Access. However, setting this up would mean that we are essentially create an instance that would be used as a proxy to send traffic from Cloud Run to the Google Compute instance.
Over the recent weekends, I’ve decided to take a gander and try another “serverless” tool called Google Cloud Workflows. The tool’s appeal is to be able coordinate a bunch of services in order to achieve a particular goal. The coordination effort (or workflow) can easily get pretty complex -> one way would be to script but if we want to have the capability to have the button to run the entire workflow from start to end with logging in place as well as capability to run the workflow based on particular triggers.
There are various serverless compute solutions on the Google Cloud Platfrom; initially it used to be only Appengine and Google Cloud Function. Google Appengine is a solution that allow you to focus on writing up apps and allow Google to take of deployment/scaling/operations. Google Cloud Functions take a step further and allow you as a developer to develop just plain old functions and allow Google to handle the rest of it, thereby making it easier to split your app functionality to parts that require to scale and parts that don’t need to.
Disclaimer: There are definitely better ways of doing this; this is more of a lazy man’s way of doing it. This is just to explore the possibility of getting a golang application into AWS Lambda and successfully running it.
Following from the previous blog post: Using AWS Lambda for Data Science Projects and Automations - Part 1
Let’s deploy a serverless application!
Problem Statement:
The application we would be trying out this time will do the following:
A thought experiment # Let’s say there was this one day during your usual work hours where you are tasked to handle some data transformations between your data sources. The data source is csv file generated from backend systems and is provided on the hourly basis. These data sources are to be analyzed as soon as possible and the insights are to be relayed to the marketing and business intelligence teams. How should we handle this? (Of course we should aim for as cheap a solution as possible)