While working on a couple of projects that would be deployed on Google Cloud Run, I realized that a couple of them tend to have some sort of similar structure. Due to the number of repositories I would typically handle on a personal basis as well as the amount of context switch I would need to move between projects; it would ideal that all of such projects are automated as much as possible.
A classic move to reduce the attack surface of Google Cloud Instances is follow the advice below:
If service on instance don’t need Public IPs, don’t attach Public IPs to such instances If instance requires Public IPs, ensure that only specific ports that are required are exposed. Clamp down on the rest of the ports and ensure no ingress on them With these basic principles, it would be simple to think how these would eventually lead to an architecture where users access the instances via a bastion host. A bastion host is a instance that would allow user to ssh in from the “outside” world. The more critical instances would linked together in a private network that is unaccessible from the outside (except for load balancers to receive traffic etc).
There are various serverless compute solutions on the Google Cloud Platfrom; initially it used to be only Appengine and Google Cloud Function. Google Appengine is a solution that allow you to focus on writing up apps and allow Google to take of deployment/scaling/operations. Google Cloud Functions take a step further and allow you as a developer to develop just plain old functions and allow Google to handle the rest of it, thereby making it easier to split your app functionality to parts that require to scale and parts that don’t need to.
So recently, I’ve been needing to automate my builds for my few Golang projects via Google Cloud Build. However, rather than building docker containers, I needed Golang binaries instead, which kind of meant that I would need to have the CI/CD pipeline have a Go environment/runtime to build them. However, when it comes to these CI/CD solutions, including private Golang packages/modules in siad projects is usually quite troublesome. Private Golang packages usually take the code from private Github/Bitbucket/Gitlab repos and getting the go get command to fetch them successful require a bit of hacks here and there to make it work successfully.
As one writes several python applications to be targeted on the Google Cloud Functions platform, it becomes increasingly obvious to pull out the more common bits of code out into its own library. Let’s have an example on the reason for this.
Recently, Google has been launching a couple of certification programs that would help people dictate their knowledge and expertise levels with the Google Cloud Platform. At the moment (January 2019), there are 7 certifications including Professional Cloud Architect, Cloud Developer and Professional Data Engineer.
There are various tooling out there to make deployment of applications easier. Some tools are used in order to help developers and organizations attempt to reach the “12 factor app” standard of applications which are set of applications that are explicitly designed to be able to scale where needed.
The list would be updated as time goes by in the year
A list of conferences and meetups and exhibitions to look for especially in 2019:
This kind of personal list that I’m keeping track; it mainly revolves around Golang, modern architecture technologies e.g. Cloud technologies etc, Python and even R (One of my initial language, I still do keep a lookout of how it’s doing nowadays.)
This is a continuation of previous blog post.
To summarize the previous related blog post.
Too painful to have people respond and react to report generation and compilation Too expensive to have machine lying around to pick up the slack and automate the reports; serverless solutions (pay on use) could be a useful model to use when running automated reports. Scenario presented for example purposes: 3 reports generated which are to be compiled to a single report. Previously mentioned 3 reports would be processed on the condition when the data files are dropped into the storage buckets. Event generated from it would automatically run the report Compilating reports # The next part of resolving our above mentioned situation (read previous blog post - part 1 for more details on this) is to compile the report. There are several ways to handle, each with their own advantages and drawbacks respectively. We would use the terms subreport to refer to reports for the initial set of reports that would then need to be compiled into a final report. These are just possible solutions; the combination of products that can be used to achieve the final goal of checking subreports and then compiling into the final report.
Seeing how functions change the way one looks at compute workloads in terms of products makes me wonder how one/companies can look at their analytics workloads and try to see if it was possible to change the costing model in that direction.
Data engineering work usually serves to be fundamentally one of the important bits when it comes to report generation in the business. The act of connecting of understanding the data that goes through the business and the need to maintain all the scripts that handle the pulling and merging all of such data makes the job way harder than one can expect. You are not expected to just be a script junkie; you are expected to be an expert at your domain, understanding the different nuances and assumption each line of script imposes on the processing of such data.
This is a little experiment to see how this would work; in the case where we have multiple Go binaries with multiple web applications. If we wanted to expose this via a single http endpoint rather than providing a whole multitude of web endpoints.
Git is one of the most important tools in a software developer arsenal. It is one of the main tool developers use in order to handle and control their code versioning. Mastering it would definitely make one’s life way easier and better; failing to do so will bring one into a world of pain. This post doesn’t intend to explain vital concepts such as git branches and forks and remotes in great detail so it would ideal if one pick those conecepts before proceeding on with the commands.
This post details my naive attempt to bring up a Kubernetes cluster on a VM. These steps to try out Kubernetes in a bare Google Virtual Machine (but the following steps should work for most Debian/Ubuntu virtual machines). This deploys a single node Kubernetes cluster (naturally don’t think of using this for production)
Meetup.com is a pretty nice site to setup meetups and sharings on technologies. The platform is pretty nice and easy to use when it comes to bookings but sometimes, the data provided by its web interface is not sufficient nor does it fit our use case. In this case, let’s say you are trying to understand the trend of the number of people attending a meetup. To an organizer, an important thing to him/her is to understand what kind of actions would lead to higher turnups/registrations for a meetup. So, by the end of this post, hopefully we would be able to have a pretty decently priced (free if possible) solution for an analytics solution which would only be called occasionally.
When developing application that are meant to be deployed to the Kubernetes platform, it involves a bunch of steps on top of your usual local development work:
Writing a Dockerfile to package the application (Multi stage applications are optional here - useful for compiled based languages) Build and tagging the docker image of the application with the target repository Either use kubectl commands or use kubernetes config resource files to define the resources required for deploying the applications. Use those commands/configurations to define the resources on the staging/production application Repeat the process for each update of the application (Repeat second point onwards) As you see from above, it starts to be pain to do so after each iteration of the application development. The building of the docker containers process as well as the applying of the new images to each cluster, (sometimes with slightly changed configuration files) - the kubernetes secret and config files can change across different environments.
There is an interesting Javascript project that allows one to use just plain old text and convert those said text into diagrams.
graph TD Start --> Stop Text to that converts to the above diagram:
Google recently had a Google Cloud conference where they feature all the exciting new technologies that are made available in Google Cloud. There are numerous product announcements that were mentioned in the various keynotes as well as the breakout sessions; so in an attempt to understand what’s exactly is happening in terms of major product releases, I did a textual overview of the videos with links to additional resources. This is to highlight some of the brand new google products and how they can be used to serve new business needs and capabilities.
This is not a in depth summary of the talks in Google IO Extended 2018. Rather, it is my notes from attending to the conference, which are heavier on links to find more about the topics. There are 2 days to the IO Extended 2018 event in Singapore. The list of talks below are from the second day.
This is not a in depth summary of the talks in Google IO Extended 2018. Rather, it is my notes from attending to the conference, which are heavier on links to find more about the topics. There are 2 days to the IO Extended 2018 event in Singapore. The list of talks below are from the first day.