Providing Aws Msk Kafka Access To External Services

Setting up private Kafka on AWS is quite straight from the console. Which works well if all your services that depends on Kafka are on same VPC. But if you want to push events from a third party hosted application like Rudderstack events to Kafka as destination, which requires you to provide SASL auth details, it becomes a bit complicated. This article aims to document the process.

If don’t already have a MSK cluster created, you can use following Terraform script using this module to create one

# main.tf
module "external_msk" {
  source = "github.com/krsoninikhil/terraform-modules//msk"

  name          = "external-cluster"
  make_public   = var.msk_make_public
  vpc_id        = var.vpc_id
  no_of_nodes   = 2
  instance_type = "kafka.t3.small"
  scram_users   = [{ username = "admin", password = "password" }]
}

# terraform.tfvars
msk_make_public = false

SASL is authentication framework and AWS MSK supports multiple such mechanism — SCRAM (username password based authentication) using AWS Secret Manager for managing credentials, IAM Authentication (using IAM users and policy instead of Kafka ACLs) and TLS based auth. You can allow public access to your MSK cluster by following steps from this doc:

• Turn off the plaintext communication, public cluster needs to have TLS enabled
• Turn on the SCRAM and IAM based auth and turn off the unauthenticated access
• Create secrets with username and password follwing this doc and attach to your cluster
• Ensure relevant ports are open for connections - 9198 for IAM auth and 9196 for SCRAM
• Turn on the public access

If you used above Terraform script, just change the msk_make_public variable to true and apply the config again. Note that MSK does not allow public access during creation, so this needs to modified by applying it again after the variable change.

💡 Gotcha: While creating credentials in Secret Manager for your cluster, you must use “Plaintext” and not the key/value editor, even though it generates the same text. I’m still not why key/value method does not work properly, if you do, please let me know.

That’s where the doc ends, so it should be done here? Let’s try to connect the brokers via kcat

$ kcat -L -b "broker1:port,broker2:port" -t "test_topic" -p 1 -X security.protocol=SASL_SSL -X sasl.mechanism="SCRAM-SHA-512" -X sasl.username=<user-from-secret> -X sasl.password=<password-from-secret>

Delivery failed for message: Broker: Topic authorization failed

Failing authorization because the SCRAM user we just added will not have appropriate ACL or permissions setup to access or create the topic. For MSK with public access, the allow.everyone.if.no.acl.found property must be set to false, so it must need ACLs. This doc explains how to create ACLs using kafka client scripts that comes with kafka installation. If kafka is not installed, you can use a docker container for kafka.

$ docker run --rm confluentinc/cp-kafka:latest /bin/sh \
  /bin/kafka-acls --bootstrap-server "broker:port"  --command-config config.properties --add --allow-principal User:<kms-user> --operation Read --topic test_topic

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=superbio_ui_events_staging, patternType=LITERAL)`:
    (principal=User:gossupkafkaprod, host=*, operation=READ, permissionType=ALLOW)

Error while executing ACL command: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createAcls
... 

If you think about this, this was not supposed to work anyway, it’s trying to give access to the self. The problem is we made the MSK public before creating the ACL, this could have worked if executed from same VPC as MSK before making it public. Now, we can’t use SCRAM auth for creating ACLs. It can done however by using IAM auth since MSK ingores the ACLs if authenticating through IAM. So we can use IAM auth to create topic publish and read ACLs and then use SCRAM with external services like Rudderstack destination.

So, create an IAM user with appropriate policy as mentioned in this AWS doc and run following Go code with the above created IAM user creds to create a topic with read write ACL on it for the given SCRAM user. Since the TLS in transit is enabled in public clusters, Go will need the CA certificate to trust, it’s not required in Java world as it already has a trust store. You can get the certificate using this command, you can use the last certificate in the listed chain.

$ openssl s_client -showcerts -connect broker1:port

Topic creation and permissions can be verified using kcat

$ kcat -L -b "broker1:port,broker2:port" -t "test_topic" -p 1 -X security.protocol=SASL_SSL -X sasl.mechanism="SCRAM-SHA-512" -X sasl.username=<user-from-secret> -X sasl.password=<password-from-secret>

Metadata for all topics ...
2 brokers:
...

Summary

• Create MSK cluster and then make it publicly accessible. You can use above Terraform config to create if not already created and then update it to allow public access.
• Create IAM user which appropriate policy to allow creating topic and publish
• Use above Go code to create topic and ACL for the provided SCRAM user
• Now, you can use this user with external services

Discussion

• Reddit post

– Nikhil

Getting Started With Music Theory

Introduction to Music - Part 1

Last month I tried understanding how music is written and communicated and I have to say, once you get to know the basics of music theory, the appreciation of good music increases exponentially. You can see what the musician just did that made you like it so much, why guitar players make weird finger pattern while playing. This is me just logging my understanding and taking notes to refer later on. If you understand music, do share your feedback.

Introduction

Musician figured out that sounds of what frequencies when played together or in a sequence with particular interval sounds good and what doesn’t. Most of the western music is created by dividing a range of frequency called an octave in 12 intervals of a fixed ratio defined by notes. Once the ratio of intervals between the notes is defined, same ratio can give you notes of higher or lower octave depending on where you start. So, if we start at 220Hz, we have one octave from 220 to 440Hz divided into 12 intervals by notes to reach 440Hz. And the next octave can start with 440Hz to 880Hz.

Piano or a keyboard is the best instrument to visualise this. Each key is a note and 12 keys (7 white, 5 black) keep repeating themselves from lower frequency octave to higher. You can go into how those interval ratio are decided but that would be too deep for this post.

Source: http://www.piano-keyboard-guide.com/wp-content/uploads/2015/05/piano-keyboard-diagram.gif

Glossary

Octave: A range of frequency divided into 12 fixed ratio notes.

Note: Sound of a particular frequency, e. g., 220 Hz. (A to G)

Accidentals: Some of notes among 12 are called accidentals and named Sharp or Flat based on the notes preceding them or following them. On a keyboard, they are represented by black keys.

Semi Tone / Half Step: A semi tone (UK) or a half step (US) is a jump or interval of 1 note, e.g. half step from C would take you to C# and half step from C# would take you to D.

Full Tone / Tone / Whole Step / Step: A whole step is a jump of 2 notes, e.g. a whole step from C would take you to D (skipping a c# in between) and from E to F#.

Key and Scale: Not to confuse with keyboard key, in music theory, key is a note on which the pattern i.e. the next set of notes, is defined. And the pattern is called the scale. Most popular scales are major, minor, diminished and augmented. For example, C Major would mean a major scale pattern or sequence of notes based on key of C.

Interval: Two notes from a scale or sequence represents an interval, e.g, 2nd interval would be 2nd note of the scale, 3rd interval would be 3rd note of the scale with respect to the first note called root. Interval is relative, so if root is moved up by a step, then 2nd, 3rd and all other intervals would also move up by a step.

Chord: Three or more notes from a sequence of notes (scale) played together is a chord.

Triad: A **Chord with exactly 3 alternate notes from a scale, i.e., root - 3rd interval - 5th interval

Degree of Chord: Degree defines where to start the chord on a scale, e.g. 3rd degree triad of a C major scale would start from 3rd note of the scale and contain 5th and 7th note.

Melody: Chords played along with other notes.

Notes

Music theory divides these notes as pure and accidental. Seven Pure denoted by English alphabets - A, B, C, D, E, F, G and five accidentals denoted by # (sharp) or b (flat) relative to pure note e.g. note between A and B is called A# or Bb. In keyboard, white keys are pure notes (repeating from A to G) and black keys are accidentals notes represented in terms of pure note so, black key between C and D can be called C# or Db. So 12 notes would be — A, A#/Bb, B, C, C#/Db, D, D#/Eb, E, F, F#/Fb, G, G#/Cb. Notice that there is no B# and E#, this is the way intervals are defined by musicians, if you find out the reasoning, I would love to hear it 🤔.

In keyboard, the accidentals are fitted in between pure keys while guitar doesn’t have that distinction, this might have to do something the way instruments are played i.e. to make the whole octave reachable for one hand 🤷.

Major and Minor Scales

Musicians have defined a scales as sequence of notes or relative steps from a key. For Major scales of a key, steps are defined as - Whole Whole Half Whole Whole Whole Half. So a C Major scale would be, referring keyboard image above while taking the whole or half step — C D E F G H A B, i.e. all white keys of the keyboard.

Source: http://www.piano-keyboard-guide.com/wp-content/uploads/2015/05/major-scale-formula.png

Similarly, for a minor scale of a key the sequence of steps is defined as — Whole Half Whole Whole Half Whole Whole. So the notes of C Minor scale would be C D Eb F G Ab Bb C. Notice that 2nd, 4th and 5th intervals have the same note for major and minor scale.

🎵 Mojor Scale Steps - W W H W W W H
   Minor Scale Steps - W H W W H W W

A chord on a scale can also be defined as major or minor depending on which scale the 3rd interval is falling, as the 5th is going to be same in major or minor. So 4th degree chord or triad on C Minor scale is F Ab C which would be called a F Minor chord (Fm) as it starts with F and 3rd is falling in minor scale of F and 2nd degree triad on C Major scale is D F A which would be a D Minor (Dm) as it starts with D and 3rd is falling on minor scale of D.

💡 Now, since 5th is same in a major and minor scale and 3rd of a minor scale is 3 half steps away from root and 3rd of a major scale is 4 half steps away, you can get a major triad by taking 4 then 3 half steps from root and minor triad by ?> taking 3 then 4 half steps.

💡 In other words, you can get a minor from a major chord by lowering a half step on the 3rd interval and minor chord by raising the 3rd interval by a half step. For example, Dm is D F A and D major chord (written as just D) would be D F# A.

Degree and Chords Notation

The degree of the chord is represented by a roman number and if it’s major then the capital roman number while minor chord would be written as small roman number.

For example, we’ll write chords or triads (root, 3rd and 5th) of C Major scale (C D E F G A B C) with different degree. Remember, If the chord has 3rd and 5th with 4 steps then 3 steps from root, it’s major chord and minor if 3rd and 5the are 3 step then 4 step away from the root.

7th degree triad gives something called a diminished chord which is denoted by a naught superscript to the roman number (vii), we’re ignoring this for now. And similarly, 1-7 degree chords of A Minor scale (A B C D E F G A) would be - Am, B diminished, C, Dm, Em, F, G and denoted as i, ii, III, iv, v, VI, VII.

Not that we’re clear of popular pattern of frequencies that are used to play music, there is time aspect to it as well. The series of notes follows a certain rhythm or time gap between the tones, understanding that would allow to communicate the speed of music notes in which they should be played.

Rhythm

Rhythm guides the music notes with respect to time, i.e., when to play the subsequent notes of a pattern and for how long before playing another note or taking rest. It gives a pattern of tone and rests in terms of beat. The basic unit of rhythm can be set against time by setting beats per minute (BPM), so same notes can be played slow or fast in time but the rhythm remains same. For example, 30 bps would mean 1 beat every 2 seconds while 60 bps is 1 beat per second. 4 beats makes a measure or bar and represents a whole note so if a note is played every beat then it’s quarter note i.e. 4 notes in a measure. If 2 notes are played with every beat i.e. 8 notes per measure then that would be a eighth note. And if there are 4 notes every beat, then it’s a sixteenth note and a measure would have 16 notes. You can keep dividing the beat, but these are the most popular ones.

To count or write notes in time i.e. beat, popular notation is -

Reference - Found this explanatory video by Andrew Huang very helpful.

– Nikhil

Confusing Terms In Container Ecosystem

Containers are great, in fact, software components which enable them are some of my favorite projects. But the more you try to know about how they work, the more terminology gets confusing. Docker, Docker Engine, containerd, runC, rkt, cri-o – all describing themselves are container runtime, are they alternatives or complementary components? Based on my understanding while trying to contribute to one of these projects, I’ve summarized these terms below. This assumes you’ve used containers before. If you have any suggestions, please let me know.

Going bottom-up, Open Container Initiative or OCI is a project by Linux Foundation which standardizes the container runtime, image and distribution specifications, these specifications are released as runtime-spec, image-spec and distribution-spec respectively.

runC is one of the many implementations of runtime-spec, other implementations can be found here. Now given the specs, these implementations have very limited and defined scope in container world i.e. runc can create, start and delete a container. Generally, a more comprehensive runtime will be required in real systems and can be implemented on top of runC. This runtime would be responsible for managing multiple containers at a time and things like downloading container images, managing storage and network interfaces, etc. Widely used example of this runtime is containerd which can manage multiple containers using runC or any other OCI implementation. Check this post by Michael Crosby to know more about how containerd integrates runC.

Another runtime is cri-o by Kubernetes which is OCI implementation with just enough functionality that is required by Kubernetes CRI. CRI or Container Runtime Interface is a Kubernetes interface for using any runtime with its container manager called kubelet. That means, containerd and cri-o are alternative runtimes for using with kubelet.

Now, kubelet and Docker Engine are a higher level abstractions that use and depend on above-mentioned runtimes. Docker Engine uses containerd and adds things like networking, volumes and security to the containers.

Following image shows how these pieces fit together visually.

Source: https://blog.docker.com/2017/08/what-is-containerd-runtime/

Docker, the company, also provides an end-user CLI client docker to interact with Docker Engine and what we use directly on our local systems to manage containers. Don’t be confused if Docker, containerd and runC all are described as runtimes, they are classified as runtimes but with different and very defined scopes.

Another interesting project is the Moby Project, which you might have come across if you’ve been reading about these components. So, earlier Docker used to be a huge piece of software by Docker, the company. With time, they separated out the major components like containerd which can be used and developed independently. All these components that Docker now uses as upstream along with some tools and framework to assemble them, comes under this Moby Project. This introductory post talks about the same in detail.

Another parallel container project is called rkt, which implements a whole different approach to manage containers and developed by CoreOS. It has a runtime which supports OCI images but as of now, it does not follow exact runtime-spec. Think of rkt as an alternative to Docker Engine along with containerd and runC. rkt also has CRI implementation for using with Kubernetes, called rktlet. Following image captures this comparison along with other details.

Source: https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html#rkt-vs-docker

To summarize, runC is OCI spec implementation for creating, starting and deleting containers. containerd uses runC and provides a broader runtime for containers. Docker client uses Docker Engine on local systems, which is built on top of containerd. Docker uses Moby Project as upstream which includes all these components. Kubernetes can use containerd, cri-o or rktlet as it’s container runtime. rkt is an alternative to Docker and has all components of its own.

[Edit]

1. Adding about LXC and libcontainer, they both use kernel features like namespaces and cgroups to provide the virtualization and have different motivations. Docker Engine used to use LXC before they created their own libcontainer. So, runC is mostly just a wrapper around libcontainer.

2. Fixed broken image links.

Reddit discussion link.

PS: Thanks Fakabbir Amin for reading the draft of this post.

A Quest To Find Perfect Editor

I used to use Atom, it’s MIT licensed, got tons of extensions. I was happy, life was good. Then Microsoft baught Github. Soon enough, people started talking about future of Atom. Since they already have something similar called VSCode, I too started worrying about their long term commitment to Atom. Few of my friends use VSCode and I’d heard and seen some very good things about it. Now that both were owned by Microsoft, there was no reason to stick with Atom and not give VSCode a try. So I did. It turned out great. Extension support and community is so great that I installed extension for almost all languages I used to write in that time. I started using it as my main editor and kept searching for something that isn’t owned by MS. I wanted something I can trust to take it to my grave. Options were very clear – Vim or Emacs?

I had tried out these editors before and they require time and effort to get used to, which I had never invested. But, it was never Vim vs Emacs for me. Full disclosure, during my junior year in college, I got a chance to attened a talk by Richard Stallman and I was already influenced by FSF to some extent. After that talk, I tried quiting Facebook (which I eventually did), reading about life without Google and using Emacs as default editor. None of them actually worked out that time.

So now that I was looking for an editor, I reconsidered Emacs, until I accidently pressed some keys (C-x m) and a mail client like buffer opened up. I was already overwhelmed by it and things like these got me off. I just wanted an editor where I can code and not a mail client (it isn’t). That day, I started setting up Vim.

Soon enough, I got used to selecting and deleting words with ved and moving around line like 0 and $ and not being able to use these keys when writing mails or anything that involves typing or editing a little more than a word or two, turned out be an irritating point. That mail client looking like Emacs buffer started to make sense now. An editor that once you get used to will be there for you, whenever or wherever you need to type a word. Need a REST client, want to search through a command’s output or may be just want to write some plain English, it got you covered. Sure, Vim might also provide those things, but I moved back to Emacs.

If you want to move to Emacs, here is my take on setup to smoothly transition from your current editor.

– Nikhil

Easy Moving From Vscode To Emacs

If you have already decided to go with Emacs as your choice of editor, congratulations! This is going to be great. However, the part after this is what’s hard and takes time. You might already be missing some of the very basic functionalities like opening a project file without typing the full path or searching something project-wide that you are very accustomed to. These features usually comes out of the box or by installing some extensions that do not require any configuration, with most of graphics based editors like Atom, VS Code or Sublime. If that is the case, keep reading. When I first started using Emacs, I spent quite some time reading and trying out stuffs to make these things work and this article is exactly because of that – to make your transition smoother. The focus is NOT on being more productive with Emacs or to give you tricks and fancy lisp code that will make you Emacs ninja overnight (they exists btw). That is something you can focus on once you’re comfortable using Emacs. This is my attempt to make every tyro feel home, to give you a setup that will allow you to do all the stuff that you’re used to.

Emacs is built around the philosophy of extensibility and flexibility – being able to hook your own code at any event of your choice is what makes it awesome. And because of that, the huge community around the editor have highly opinionated setups based on what works for them. There exists modified versions of Emacs like Prelude, Spacemacs, etc. that are loaded with a lot of functionalities you wouldn’t find anywhere else but in my personal opinion, it’s good to know what is cake and what are the cherries.

Before we start, familiarize yourself with basic key combinations and terminology of Emacs. Built-in tutorial explains that very neatly which you can read by C-h t (hold control key and press h, then leave control key and press t). This awesome article from last week’s HN front page is also a great read for this.

Let’s start. After moving from Atom and VSCode to Emacs, I missed these functionalities:

1. Being able to duplicate current line
2. Move or drag a line up and down
3. Use multiple cursors to edit at a time
4. Auto-completion of variables and functions name
5. Fuzzy file search i.e. Ctrl-P or Cmd-Shift-o in other editors
6. Searching a term of string in the whole project

7. A tree view of the project to explore file structure

   Few more things that I had installed extensions for in VSCode:

8. Goto functions and class definitions
9. Come back to previous cursor position
10. Find all the reference of a function
11. Markdown preview

Surprisingly, some of these are as easy as hooking a couple of lines of code to a keystroke, for some a little more than a couple of lines so we’ll use community developed packages for them. The latest version of Emacs ships with a package manager – package.el. We just need to add MELPA repository to it so it can search packages for us. Add this Emacs Lisp (ref) to your ~/.emacs.d/init.el file to do that.

(require 'package)
(setq package-archives
      '(("GNU ELPA"     . "https://elpa.gnu.org/packages/")
        ("MELPA Stable" . "https://stable.melpa.org/packages/")
        ("MELPA"        . "https://melpa.org/packages/"))
      package-archive-priorities
      '(("GNU ELPA"     . 10)
        ("MELPA Stable" . 5)
        ("MELPA"        . 0)))
(package-initialize)

This will also give priorities to repositories to avoid duplicate listing in case of certain packages being present in more than one. After adding this restart Emacs to reload init.el and do this: M-x package-refresh-contents. Now you are ready to install most of the packages. Note that you can also reload you init.el by doing M-x load-file and then providing file path.

Rest of the article is about enabling above mentioned functionalities one by one.

1. Add this code to same init.el

    (defun duplicate-line ()
      (interactive)
      (save-mark-and-excursion
        (beginning-of-line)
        (insert (thing-at-point 'line t))))
   
    (global-set-key (kbd "C-S-d") 'duplicate-line)
   

   Now, you can use C-S-d (Control-Shift-d) to duplicate current line. Change these key combination to whatever suits you.

2. Add these lines for using C-S-j and C-S-k to move a line up or down one line.

    (defun move-line-down ()
      (interactive)
      (let ((col (current-column)))
        (save-excursion
          (forward-line)
          (transpose-lines 1))
        (forward-line)
        (move-to-column col)))
   
    (defun move-line-up ()
      (interactive)
      (let ((col (current-column)))
        (save-excursion
          (forward-line)
          (transpose-lines -1))
        (forward-line -1)
        (move-to-column col)))
   
    (global-set-key (kbd "C-S-j") 'move-line-down)
    (global-set-key (kbd "C-S-k") 'move-line-up)
   

3. To get mutiple cursor, I use multiple-cursor package, which can be installed by M-x package-install <RET> multiple-cursor <RET>. Add these key bindings to easily use it.

    (require 'multiple-cursors)
    (global-set-key (kbd "C-|") 'mc/edit-lines)
    (global-set-key (kbd "C->") 'mc/mark-next-like-this)
    (global-set-key (kbd "C-<") 'mc/mark-previous-like-this)
    (global-set-key (kbd "C-c C-<") 'mc/mark-all-like-this)
    (global-set-key (kbd "C-S-<mouse-1>") 'mc/add-cursor-on-click)
    (define-key mc/keymap (kbd "<return>") nil)
   

   These commands do exactly what they look like. To get out of multiple cursors, use C-g, the last line in above code prevents using <RETURN> key to do same. You can read more about it in the project doc page.

4. To let Emacs auto-complete function names and variables, I use company, which can be installed by M-x package-install <RET> company-mode <RET>. And can be activated by adding this line to init.el:

    (add-hook 'after-init-hook 'global-company-mode)
   

5. I use the combination of projectile and helm package to deal with fuzzy file search. Configurations for helm can be a bit overwhelming so just start with following.

    (require 'projectile)
    (setq projectile-indexing-method 'alien)
    (setq projectile-enable-caching t)
    (projectile-global-mode)
   
    (require 'helm)
    (require 'helm-config)
    (global-set-key (kbd "C-c h") 'helm-command-prefix)
    (global-unset-key (kbd "C-x c"))
    (helm-autoresize-mode 1)
    (global-set-key (kbd "M-x") 'helm-M-x)
    (setq helm-M-x-fuzzy-match t)
    (global-set-key (kbd "C-x C-f") 'helm-find-files)
    (helm-mode 1)
   

   This will allow you to use C-c p f for opening your project files just by typing file name and not the whole path. And when you use C-x C-f to open a file that is outside the project directory, you can type any part of the directory or file name or even non contiguous parts separated by space to narrow down the suggestions e.g. if you want to select controllers out of controllers and contol folder name, you can type co s. Use C-j (not TAB) to choose highlighted option.

6. projectile can also be used to search for something in the whole project: C-c p s s. This command used ag system package, which should definitely have and a ag.el Emacs package. I don’t generally use project wild search and replace but if that’s your thing, this answer from Stackexchange explains a great way of doing so.

7. Built-in speedbar provides good enough interactive tree view of project. It can be started with M-x speedbar. It’s easier to bind it to some key like F8 using

    (global-set-key (kbd "<f8>") 'speedbar)
   

8. I use dumb-jump package to jump to definitions using C-M-g. Add this line to auto enable it every time:

    (dumb-jump-mode)
   

9. You can jump back to function call by C-M-p.

10. To find all the references of a function, priviously mentioned project-wide search with projectile works well for me.

11. I use fymd package to real time markdown preview, with this key binding:

    (global-set-key (kbd "<f9>") 'flymd-flyit)
    

Most of the snippets on this page are taken from emacsrocks.com, whattheemacsd.com and official documentation of mentioned packages. Both of these resources are great places to learn more about Emacs. Here is my .emacs.d directory, if don’t feel like doing this all by yourself, feel free to clone mine. I have all the defuns defined in ~/.emacs.d/user-lisp/defuns.el file which is imported in init.el by:

(add-to-list 'load-path "~/.emacs.d/user-lisp")
(require 'defuns)

I keep all the references to these defuns in init.el so all the key bindings remain in one file.

I would highly appreciate any feedback you may have or to listen about any other features you miss from your old editor and should be in this list.

[Edit]

Code snippet for duplicating line (first point) has been updated with current cleaner version suggested by Philip K., earlier it was:

(defun duplicate-line ()
   (interactive)
   (let ((col (current-column)))
     (move-beginning-of-line 1)
     (kill-line)
     (yank)
     (newline)
     (yank)
     (move-to-column col)))

– @krsoninikhil