Fastapi Like Controllers For Gin Based Go Apps

Everytime I start a new project in Go, setting up the boilerplat is hassle. Unless you add your configuration loader, DB connections and authentication APIs, middleware, it’s same story everytime. While writing all of these packages is fun and gets you in the flow, it takes time and it’s not as quick as writing a app in FastAPI.

The biggest pain point I faced is writing handlers and repeating request parsing logic for handler with a different request types because frameworks like Gin provides you context as argument to handler and expect you bind it however you expect which is great but Go supports generic and I don’t see why this can’t be generalised and make the request type directly available in the handler.

Getting Rid Of Request Binding In Every Handler

Gin’s handler function:

type HandlerFunc func(c *gin.Context)

So, this time when I started a personal project, I changed the focus to solve for generic based handlers. After a few iterations following seems to take care of exactly what I was looking for:

type bindedHandlerFunc[Req, Res any] func(*gin.Context, Req) (*Res, error) 

This allowed to change your route mapping from:

// request type
type (
    RequestBody struct {
        JsonField  string   `json:"json_field"`
        QueryParam string   `form:"query_param"`
    }
    ResponseBody struct {
        QueryParamInput   string `json:"query_param_input"`
        JsonFieldInput    string `json:"json_field_input"`
        PathParamInput    int    `json:"path_param_input"`
    }
)

// handler
func myHandler(c *gin.Context) {
    var req RequestBody
    if err := c.ShouldBindJSON(req); err != nil {
        return errors.Wrap(err, "error binding request body")
    }
    if err := c.ShouldBindQuery(req); err != nil {
        return errors.Wrap(err, "error binding query params")
    }

    p := c.Param("path_param") // this could be part of request struct, but then you still need to add a ShouldBindUri block
    pInt, err := strconv.ParseInt(p, 10, 64)
    if err != nil {
        return errors.Wrapf(err, "error parsing path_param: %v", p)
    }

    return c.JSON(http.StatusOK, ResponseBody{
        QueryParamInput: req.QueryParam, 
        JsonFieldInput:  req.JsonField,
        PathParamInput:  pInt,
    })
}
// routes
r.GET("/path", myHandler)

to something like:

// request type
type RequestBody struct {
    JsonField  string   `json:"json_field"`
    PathParam  int      `uri:"path_param"`
    QueryParam string   `form:"query_param"`
}

// handler
func myHandler(c *gin.Context, req RequestBody) (*ResponseBody, error) {
    return c.JSON(http.StatusOK, ResponseBody{
        QueryParamInput: req.QueryParam, 
        JsonFieldInput:  req.JsonField,
        PathParamInput:  req.PathParam,
    })
}

r.GET("/path", request.BindAll(myHandler))

Pretty neat, han?

If you don’t want to allow binding from URI and body in the same request, there request.BindGet, request.BindCreate, request.BindDelete to only bind for query, request body and path param respectively.

You can find this request package here on GitHub.

Exposing ready made CRUD Endpoints

Once this was achieved, I remeber from my Rails experience, for a CRUD api, all you need to do is provide a resource and CRUD is exposed. So to achieve similar feature, I wrote a crud package which allows you to expose the CRUD endpoints just by writing the model and then request, response type. Read more about it in the readme here - https://github.com/krsoninikhil/go-rest-kit

The module has other useful packages like pre written handlers for auth, configuration loader, postgres connection, etc.

– Nikhil

Providing Aws Msk Kafka Access To External Services

Setting up private Kafka on AWS is quite straight from the console. Which works well if all your services that depends on Kafka are on same VPC. But if you want to push events from a third party hosted application like Rudderstack events to Kafka as destination, which requires you to provide SASL auth details, it becomes a bit complicated. This article aims to document the process.

If don’t already have a MSK cluster created, you can use following Terraform script using this module to create one

# main.tf
module "external_msk" {
  source = "github.com/krsoninikhil/terraform-modules//msk"

  name          = "external-cluster"
  make_public   = var.msk_make_public
  vpc_id        = var.vpc_id
  no_of_nodes   = 2
  instance_type = "kafka.t3.small"
  scram_users   = [{ username = "admin", password = "password" }]
}

# terraform.tfvars
msk_make_public = false

SASL is authentication framework and AWS MSK supports multiple such mechanism — SCRAM (username password based authentication) using AWS Secret Manager for managing credentials, IAM Authentication (using IAM users and policy instead of Kafka ACLs) and TLS based auth. You can allow public access to your MSK cluster by following steps from this doc:

• Turn off the plaintext communication, public cluster needs to have TLS enabled
• Turn on the SCRAM and IAM based auth and turn off the unauthenticated access
• Create secrets with username and password follwing this doc and attach to your cluster
• Ensure relevant ports are open for connections - 9198 for IAM auth and 9196 for SCRAM
• Turn on the public access

If you used above Terraform script, just change the msk_make_public variable to true and apply the config again. Note that MSK does not allow public access during creation, so this needs to modified by applying it again after the variable change.

💡 Gotcha: While creating credentials in Secret Manager for your cluster, you must use “Plaintext” and not the key/value editor, even though it generates the same text. I’m still not why key/value method does not work properly, if you do, please let me know.

That’s where the doc ends, so it should be done here? Let’s try to connect the brokers via kcat

$ kcat -L -b "broker1:port,broker2:port" -t "test_topic" -p 1 -X security.protocol=SASL_SSL -X sasl.mechanism="SCRAM-SHA-512" -X sasl.username=<user-from-secret> -X sasl.password=<password-from-secret>

Delivery failed for message: Broker: Topic authorization failed

Failing authorization because the SCRAM user we just added will not have appropriate ACL or permissions setup to access or create the topic. For MSK with public access, the allow.everyone.if.no.acl.found property must be set to false, so it must need ACLs. This doc explains how to create ACLs using kafka client scripts that comes with kafka installation. If kafka is not installed, you can use a docker container for kafka.

$ docker run --rm confluentinc/cp-kafka:latest /bin/sh \
  /bin/kafka-acls --bootstrap-server "broker:port"  --command-config config.properties --add --allow-principal User:<kms-user> --operation Read --topic test_topic

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=superbio_ui_events_staging, patternType=LITERAL)`:
    (principal=User:gossupkafkaprod, host=*, operation=READ, permissionType=ALLOW)

Error while executing ACL command: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createAcls
... 

If you think about this, this was not supposed to work anyway, it’s trying to give access to the self. The problem is we made the MSK public before creating the ACL, this could have worked if executed from same VPC as MSK before making it public. Now, we can’t use SCRAM auth for creating ACLs. It can done however by using IAM auth since MSK ingores the ACLs if authenticating through IAM. So we can use IAM auth to create topic publish and read ACLs and then use SCRAM with external services like Rudderstack destination.

So, create an IAM user with appropriate policy as mentioned in this AWS doc and run following Go code with the above created IAM user creds to create a topic with read write ACL on it for the given SCRAM user. Since the TLS in transit is enabled in public clusters, Go will need the CA certificate to trust, it’s not required in Java world as it already has a trust store. You can get the certificate using this command, you can use the last certificate in the listed chain.

$ openssl s_client -showcerts -connect broker1:port

Topic creation and permissions can be verified using kcat

$ kcat -L -b "broker1:port,broker2:port" -t "test_topic" -p 1 -X security.protocol=SASL_SSL -X sasl.mechanism="SCRAM-SHA-512" -X sasl.username=<user-from-secret> -X sasl.password=<password-from-secret>

Metadata for all topics ...
2 brokers:
...

Summary

• Create MSK cluster and then make it publicly accessible. You can use above Terraform config to create if not already created and then update it to allow public access.
• Create IAM user which appropriate policy to allow creating topic and publish
• Use above Go code to create topic and ACL for the provided SCRAM user
• Now, you can use this user with external services

Discussion

• Reddit post

– Nikhil

Getting Started With Music Theory

Introduction to Music - Part 1

Last month I tried understanding how music is written and communicated and I have to say, once you get to know the basics of music theory, the appreciation of good music increases exponentially. You can see what the musician just did that made you like it so much, why guitar players make weird finger pattern while playing. This is me just logging my understanding and taking notes to refer later on. If you understand music, do share your feedback.

Introduction

Musician figured out that sounds of what frequencies when played together or in a sequence with particular interval sounds good and what doesn’t. Most of the western music is created by dividing a range of frequency called an octave in 12 intervals of a fixed ratio defined by notes. Once the ratio of intervals between the notes is defined, same ratio can give you notes of higher or lower octave depending on where you start. So, if we start at 220Hz, we have one octave from 220 to 440Hz divided into 12 intervals by notes to reach 440Hz. And the next octave can start with 440Hz to 880Hz.

Piano or a keyboard is the best instrument to visualise this. Each key is a note and 12 keys (7 white, 5 black) keep repeating themselves from lower frequency octave to higher. You can go into how those interval ratio are decided but that would be too deep for this post.

Source: http://www.piano-keyboard-guide.com/wp-content/uploads/2015/05/piano-keyboard-diagram.gif

Glossary

Octave: A range of frequency divided into 12 fixed ratio notes.

Note: Sound of a particular frequency, e. g., 220 Hz. (A to G)

Accidentals: Some of notes among 12 are called accidentals and named Sharp or Flat based on the notes preceding them or following them. On a keyboard, they are represented by black keys.

Semi Tone / Half Step: A semi tone (UK) or a half step (US) is a jump or interval of 1 note, e.g. half step from C would take you to C# and half step from C# would take you to D.

Full Tone / Tone / Whole Step / Step: A whole step is a jump of 2 notes, e.g. a whole step from C would take you to D (skipping a c# in between) and from E to F#.

Key and Scale: Not to confuse with keyboard key, in music theory, key is a note on which the pattern i.e. the next set of notes, is defined. And the pattern is called the scale. Most popular scales are major, minor, diminished and augmented. For example, C Major would mean a major scale pattern or sequence of notes based on key of C.

Interval: Two notes from a scale or sequence represents an interval, e.g, 2nd interval would be 2nd note of the scale, 3rd interval would be 3rd note of the scale with respect to the first note called root. Interval is relative, so if root is moved up by a step, then 2nd, 3rd and all other intervals would also move up by a step.

Chord: Three or more notes from a sequence of notes (scale) played together is a chord.

Triad: A **Chord with exactly 3 alternate notes from a scale, i.e., root - 3rd interval - 5th interval

Degree of Chord: Degree defines where to start the chord on a scale, e.g. 3rd degree triad of a C major scale would start from 3rd note of the scale and contain 5th and 7th note.

Melody: Chords played along with other notes.

Notes

Music theory divides these notes as pure and accidental. Seven Pure denoted by English alphabets - A, B, C, D, E, F, G and five accidentals denoted by # (sharp) or b (flat) relative to pure note e.g. note between A and B is called A# or Bb. In keyboard, white keys are pure notes (repeating from A to G) and black keys are accidentals notes represented in terms of pure note so, black key between C and D can be called C# or Db. So 12 notes would be — A, A#/Bb, B, C, C#/Db, D, D#/Eb, E, F, F#/Fb, G, G#/Cb. Notice that there is no B# and E#, this is the way intervals are defined by musicians, if you find out the reasoning, I would love to hear it 🤔.

In keyboard, the accidentals are fitted in between pure keys while guitar doesn’t have that distinction, this might have to do something the way instruments are played i.e. to make the whole octave reachable for one hand 🤷.

Major and Minor Scales

Musicians have defined a scales as sequence of notes or relative steps from a key. For Major scales of a key, steps are defined as - Whole Whole Half Whole Whole Whole Half. So a C Major scale would be, referring keyboard image above while taking the whole or half step — C D E F G H A B, i.e. all white keys of the keyboard.

Source: http://www.piano-keyboard-guide.com/wp-content/uploads/2015/05/major-scale-formula.png

Similarly, for a minor scale of a key the sequence of steps is defined as — Whole Half Whole Whole Half Whole Whole. So the notes of C Minor scale would be C D Eb F G Ab Bb C. Notice that 2nd, 4th and 5th intervals have the same note for major and minor scale.

🎵 Mojor Scale Steps - W W H W W W H
   Minor Scale Steps - W H W W H W W

A chord on a scale can also be defined as major or minor depending on which scale the 3rd interval is falling, as the 5th is going to be same in major or minor. So 4th degree chord or triad on C Minor scale is F Ab C which would be called a F Minor chord (Fm) as it starts with F and 3rd is falling in minor scale of F and 2nd degree triad on C Major scale is D F A which would be a D Minor (Dm) as it starts with D and 3rd is falling on minor scale of D.

💡 Now, since 5th is same in a major and minor scale and 3rd of a minor scale is 3 half steps away from root and 3rd of a major scale is 4 half steps away, you can get a major triad by taking 4 then 3 half steps from root and minor triad by ?> taking 3 then 4 half steps.

💡 In other words, you can get a minor from a major chord by lowering a half step on the 3rd interval and minor chord by raising the 3rd interval by a half step. For example, Dm is D F A and D major chord (written as just D) would be D F# A.

Degree and Chords Notation

The degree of the chord is represented by a roman number and if it’s major then the capital roman number while minor chord would be written as small roman number.

For example, we’ll write chords or triads (root, 3rd and 5th) of C Major scale (C D E F G A B C) with different degree. Remember, If the chord has 3rd and 5th with 4 steps then 3 steps from root, it’s major chord and minor if 3rd and 5the are 3 step then 4 step away from the root.

7th degree triad gives something called a diminished chord which is denoted by a naught superscript to the roman number (vii), we’re ignoring this for now. And similarly, 1-7 degree chords of A Minor scale (A B C D E F G A) would be - Am, B diminished, C, Dm, Em, F, G and denoted as i, ii, III, iv, v, VI, VII.

Not that we’re clear of popular pattern of frequencies that are used to play music, there is time aspect to it as well. The series of notes follows a certain rhythm or time gap between the tones, understanding that would allow to communicate the speed of music notes in which they should be played.

Rhythm

Rhythm guides the music notes with respect to time, i.e., when to play the subsequent notes of a pattern and for how long before playing another note or taking rest. It gives a pattern of tone and rests in terms of beat. The basic unit of rhythm can be set against time by setting beats per minute (BPM), so same notes can be played slow or fast in time but the rhythm remains same. For example, 30 bps would mean 1 beat every 2 seconds while 60 bps is 1 beat per second. 4 beats makes a measure or bar and represents a whole note so if a note is played every beat then it’s quarter note i.e. 4 notes in a measure. If 2 notes are played with every beat i.e. 8 notes per measure then that would be a eighth note. And if there are 4 notes every beat, then it’s a sixteenth note and a measure would have 16 notes. You can keep dividing the beat, but these are the most popular ones.

To count or write notes in time i.e. beat, popular notation is -

Reference - Found this explanatory video by Andrew Huang very helpful.

– Nikhil

Confusing Terms In Container Ecosystem

Containers are great, in fact, software components which enable them are some of my favorite projects. But the more you try to know about how they work, the more terminology gets confusing. Docker, Docker Engine, containerd, runC, rkt, cri-o – all describing themselves are container runtime, are they alternatives or complementary components? Based on my understanding while trying to contribute to one of these projects, I’ve summarized these terms below. This assumes you’ve used containers before. If you have any suggestions, please let me know.

Going bottom-up, Open Container Initiative or OCI is a project by Linux Foundation which standardizes the container runtime, image and distribution specifications, these specifications are released as runtime-spec, image-spec and distribution-spec respectively.

runC is one of the many implementations of runtime-spec, other implementations can be found here. Now given the specs, these implementations have very limited and defined scope in container world i.e. runc can create, start and delete a container. Generally, a more comprehensive runtime will be required in real systems and can be implemented on top of runC. This runtime would be responsible for managing multiple containers at a time and things like downloading container images, managing storage and network interfaces, etc. Widely used example of this runtime is containerd which can manage multiple containers using runC or any other OCI implementation. Check this post by Michael Crosby to know more about how containerd integrates runC.

Another runtime is cri-o by Kubernetes which is OCI implementation with just enough functionality that is required by Kubernetes CRI. CRI or Container Runtime Interface is a Kubernetes interface for using any runtime with its container manager called kubelet. That means, containerd and cri-o are alternative runtimes for using with kubelet.

Now, kubelet and Docker Engine are a higher level abstractions that use and depend on above-mentioned runtimes. Docker Engine uses containerd and adds things like networking, volumes and security to the containers.

Following image shows how these pieces fit together visually.

Source: https://blog.docker.com/2017/08/what-is-containerd-runtime/

Docker, the company, also provides an end-user CLI client docker to interact with Docker Engine and what we use directly on our local systems to manage containers. Don’t be confused if Docker, containerd and runC all are described as runtimes, they are classified as runtimes but with different and very defined scopes.

Another interesting project is the Moby Project, which you might have come across if you’ve been reading about these components. So, earlier Docker used to be a huge piece of software by Docker, the company. With time, they separated out the major components like containerd which can be used and developed independently. All these components that Docker now uses as upstream along with some tools and framework to assemble them, comes under this Moby Project. This introductory post talks about the same in detail.

Another parallel container project is called rkt, which implements a whole different approach to manage containers and developed by CoreOS. It has a runtime which supports OCI images but as of now, it does not follow exact runtime-spec. Think of rkt as an alternative to Docker Engine along with containerd and runC. rkt also has CRI implementation for using with Kubernetes, called rktlet. Following image captures this comparison along with other details.

Source: https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html#rkt-vs-docker

To summarize, runC is OCI spec implementation for creating, starting and deleting containers. containerd uses runC and provides a broader runtime for containers. Docker client uses Docker Engine on local systems, which is built on top of containerd. Docker uses Moby Project as upstream which includes all these components. Kubernetes can use containerd, cri-o or rktlet as it’s container runtime. rkt is an alternative to Docker and has all components of its own.

[Edit]

1. Adding about LXC and libcontainer, they both use kernel features like namespaces and cgroups to provide the virtualization and have different motivations. Docker Engine used to use LXC before they created their own libcontainer. So, runC is mostly just a wrapper around libcontainer.

2. Fixed broken image links.

Reddit discussion link.

PS: Thanks Fakabbir Amin for reading the draft of this post.

A Quest To Find Perfect Editor

I used to use Atom, it’s MIT licensed, got tons of extensions. I was happy, life was good. Then Microsoft baught Github. Soon enough, people started talking about future of Atom. Since they already have something similar called VSCode, I too started worrying about their long term commitment to Atom. Few of my friends use VSCode and I’d heard and seen some very good things about it. Now that both were owned by Microsoft, there was no reason to stick with Atom and not give VSCode a try. So I did. It turned out great. Extension support and community is so great that I installed extension for almost all languages I used to write in that time. I started using it as my main editor and kept searching for something that isn’t owned by MS. I wanted something I can trust to take it to my grave. Options were very clear – Vim or Emacs?

I had tried out these editors before and they require time and effort to get used to, which I had never invested. But, it was never Vim vs Emacs for me. Full disclosure, during my junior year in college, I got a chance to attened a talk by Richard Stallman and I was already influenced by FSF to some extent. After that talk, I tried quiting Facebook (which I eventually did), reading about life without Google and using Emacs as default editor. None of them actually worked out that time.

So now that I was looking for an editor, I reconsidered Emacs, until I accidently pressed some keys (C-x m) and a mail client like buffer opened up. I was already overwhelmed by it and things like these got me off. I just wanted an editor where I can code and not a mail client (it isn’t). That day, I started setting up Vim.

Soon enough, I got used to selecting and deleting words with ved and moving around line like 0 and $ and not being able to use these keys when writing mails or anything that involves typing or editing a little more than a word or two, turned out be an irritating point. That mail client looking like Emacs buffer started to make sense now. An editor that once you get used to will be there for you, whenever or wherever you need to type a word. Need a REST client, want to search through a command’s output or may be just want to write some plain English, it got you covered. Sure, Vim might also provide those things, but I moved back to Emacs.

If you want to move to Emacs, here is my take on setup to smoothly transition from your current editor.

– Nikhil