Roy's musings

In 2015, I co-founded a company with my friend. Until then I had been working as an individual contributor. Though I had been a technical lead to many projects, I hadn’t managed teams. Over the past 7+ years I have learned a great deal on project and people management. My experience has taught me the most than the books and articles I have read. In this series, I would like to summarize what has worked for me with the hope that it would be helpful for someone following similar trajectory. If you have any feedback, please leave in comments. Project and people management As a manager, you are managing two things: projects and people. They both require different sets of skills.  For the most part, managing projects can be learnt from reading books. But managing people is a different skill and it cannot be learnt from books alone. It requires a lot of interacting with people. It requires listening and understanding people’s needs so that you can help them succeed. Usually these nee...

Service health checks are ubiquitous. If you have built any software that relies upon any upstream service, I am sure you would have used some form of health check. Your software could be a stand alone program or it could be a proxy. You may also have built and exposed a service that is used as an upstream service by another software. The worst way to perform health check is to pull a resource (like GET /hc.html, if your service is exposed as a HTTP service) or perform a TCP connect check in the same port where data is served . These are checks I call as in-band health checks .  Please don't do these. Based on my experience, the best way to expose health checks is by using an out-of-band mechanism. This means that you expose another port for performing health check. The client can perform a HTTP check or TCP check in the health check port. As a convenience, if you expose health check as HTTP service in health check port, you can consolidate multiple health checks like: GET /he...

There had been instances when I wanted to copy "/home/roy" to a target directory "/target" as "/target/home/roy". I used to do some scripting foo to get this done. A few days back, I learned this cool option called "--parents" in cp command that will preserve the parent directory structure while copying. Made my life a lot easier on one of my backup tasks.

Today I Learned (TIL) is a series of posts that contain bit sized information that I learned and found to be worthy of sharing with others and keep as a note to myself. For the last couple of years, I had been devoting most of my time in building the start-up, Zycada Networks , that I co-founded. It makes me happy to share my knowledge. So I would like to get back to the habit of writing blog posts regularly on interesting stories, tips-and-tricks, new ideas, technology trends, etc. TIL is one part of doing it.

Sorting numeric values in the input is easy. It just takes "sort -n". But what if the input contains human friendly units. For intance, "5G", "3M", "4K", etc. There is a flag to recognize and sort based on the human friendly units: "-h". Incorrect: du -h | sort -nr Correct: du -h | sort -rh Caveat: "sort -h" works only on upper-case units. "K" will be treated as kilos, but "k" will not be!

Aligning text like a table is often a useful task. I use this workflow to make the text tidier. For instance when I have two columns of text of varying width, aligning them makes it easier to read. Steps: 1) Select the region you would like to align (C-x h will select the entire buffer) 2) C-u M-x align Thats it. The selected region would have been aligned based on the first line of the region. Example: Before: what? who? hello      world After: what? who? hello world By default only the space is used as a delimiter. If you would like to perform a little sophisticated alignment, you can make use of the align-regex function. city, population new york, 8.49M san francisco, 0.85M

Some time back a friend asked me an algorithm question: Given an integer random number generator randomN() that can generate random number in the range [0, N), how will you generate random numbers in the range [0, M) where M I used modulo arithmetic to generate the desired random numbers in the range [0, M). And I reasoned out that if N = q*M + r, every number in the range [0, r] has an occurrence probability of (q+1)/N, but the numbers in the range [r+1, M) has an occurrence probability of only q/N. It is easy to visualize this. See the diagram below: You can see that we can divide line of length N units (given in green) by lines of length M units (given in black). When N is not exactly divisible by M, in the last part alone we have only r units. So if we choose a random integer in the line represented in green, and then take a modulo M on the value, all the values except the values in the range [r+1, M) (represented in red) will occur q+1 times. But the values in that int...

I was reviewing some piece of code and came across a function that was taking Optional  as argument. In the same class, another function was taking a couple of Optional as arguments. When I thought about it, I felt that taking an Optional should be avoided. You can return an Optional from a function, but avoid taking Optional arguments. Before calling a function, you should check for the presence or absence of the values that you are passing. Hence a function taking one or more Optional is a code smell. One argument that can possibly presented in favor of taking Optional as argument is every caller checking for the presence of the arguments. Consider the code example below: If you pay attention, the error handling from the caller's point of view is ugly if the caller wants to report correct error. Much worse, the error check is done after using the values. A side effect of product() taking Optional is that it must return an Optional. Otherwise it has to throw an ...

I am using Scala for one of my recent projects. For logging, I am using SLF4J/LOGback. There is one minor annoyance while you are trying to log two argument messages like: logger.info("Some log with arg1 [{}] and arg2 [{}].", arg1, arg2) While you compile with sbt you will get the following error: [error] both method info in trait Logger of type (x$1: String, x$2: <repeated...>[Object])Unit [error] and  method info in trait Logger of type (x$1: String, x$2: Any, x$3: Any)Unit [error] match argument types (String,String,String)   If you are getting this error, a small trick that I did to avoid casting to AnyRef or Object was to just add a null argument at the end which will force the Scala compiler to make use of the vararg version. LOGBack just ignores extraneous arguments. Like this: logger.info("Some log with arg1 [{}] and arg2 [{}].", arg1, arg2, null) Disclaimer: I am a Scala rookie, hence take my advice with a pinch of salt!  

TLDR; If you are new to IntelliJ (like me), use "Cmd+Shift+A" to find your way around. Recently I started developing an application in Scala. I am an avid Eclipse user, but unfortunately I felt that the support for Scala is a bit clunky as of today. It was driving me nuts to see some of the Java classes highlighted in red as unknown classes, but restarting the Eclipse or refreshing project was removing those errors. Based on my research, I felt that IntelliJ IDEA has better support for Scala. So I wanted to give it a spin. So far I feel so good that I decided to give it a try. I prefer to get things done using keyboard shortcuts, touching the mouse only for absolutely essential needs. The most important keyboard (and in my opinion the most awesome) shortcut that you should know in IntelliJ is "Cmd+Shift+A".  This will show you an input box where you can enter a partial string and you will be presented with the list of action or options that is relevant ...

You might have used Hibernate for your ORM needs. The most important thing that you should about Hibernate cache is that it is implemented using a Map and a reference every entity that you have read from the database is kept in that Map. The key of that Map is the Entity ID (which could be just a Long or some form of composite primary key you have defined and wrapped in an EntityKey ). The value stored is the entity itself (proxied). If you read a 1000 entities (or rows) from the database, a reference to each one of them will be kept in the Map. If those entities have relationships specified (one-to-many or many-to-many) with other entities and those entities should be loaded eagerly, you are asking Hibernate to load a lot more than 1000 entities. If you venture into processing a large set of rows (say 100,000 rows), a reference to every one those rows is going to be kept in the Map. Sometimes the amount of memory needed could be so large that your process might run out of memory ...

There was a serious vulnerability reported in the OpenSSL library that can let the attacker to dump memory contents from the server. Thus the attacker can perform offline analysis of the memory contents and identify sensitive information like private key of the server, key material for SSL sessions, decrypted data that is in memory, etc. This affects any server that uses OpenSSL to implement HTTPS. I thought I will share some material in one place that will be helpful for people to understand the problem better. Description of the problem can be found here . A simple Python script to test your servers can be found here  or you can use this site . NVD entry for this issue can be found here . How some of the companies are responding: Heroku , AWS , Lastpass . Hope that helps.

Designing a distributed system is hard enough. Even harder to design a distributed system that is reliable. There are many best practices that you can follow to make a reliable distributed system. Based on an issue that I recently troublehooted, there are a couple of them that I think are critical: Enabling TCP keep-alive between the processes if you are using TCP Performing all IO operations with a time out My advise is based on my experience in Linux. But I think it should be applicable to other operating systems as well. When a host goes down with a kernel panic, none of the established connections are closed by sending a FIN or RESET packet. This cauases trouble that the peer process doesn't know about the other end of the communication being gone. When you enable TCP keep-alive, the kernel sends a zero length packet as per the configuration. Hence if the peer has died due to kernel panic, the zero length packet will not be ACKed. Thus, the peer death is detected. If th...

One serious limitation of using Jackson to deserialize the custom defined bean classe is that you cannot differentiate between a value missing in the input JSON or it is present but having null value. Consider the exampel below: class MyBean { private String firstName; private String lastName; // Getters and setters here. } You cannot differentiate between the JSON inputs {"firstName": "John"} and {"firstName": "John", "lastName": null}. I.e. the lastName property being present in the input or not. Guava library has an excellent tool to express missing values versus null values: Optional . Jackson supports integration with Guava library. The integration library can be accessed from jackson-datatype-guava package. To clearly express the null versus absent values, we can rewrite the bean class above as follows: class MyBean { private Optional<String> firstName; private Optional<String> lastName; // Ge...

I wrote a small utility script that can be used to run a small snippet of Python code like "perl -ne". I find it very useful for my needs. Hope it helps you too. Any suggestions welcome. You can find the script as a gist here .

I had earlier written a small snippet about setting the window title from command line. Based on my experience, I felt that I could make it a lot simpler. Here is version of the same function. function wtitle {     if [ -z "$ORIG_PS1" ] ; then         ORIG_PS1=$PS1     fi     export PS1="\[\033]0;$1 - \u@\h:\w\007\]$ORIG_PS1" }

I usually do a lot of log analysis. I search for patterns in multiple files in multiple hosts. Then I collate the result and do processing on the result lines. A sample output looks like: /tmp/input1.txt: line containing pattern /tmp/input2.txt: another line containing pattern  As it turns out, I don't need the file names in the grep result. I always used to remove the file name prefixes using a Perl one-liner. Silly me! I was pretty sure that this was a common problem and it must have been solved already. When I referred to the man page of grep , I came across this gem: -h . If you specify this option, grep will not prefix each line with a the file name. There is also -H option which will prefix each line with a filename even if you are searching in only one file.

After a couple of years or so, I starting making use Python as my main programming language for one of the projects. This time I was making use of the virtualenv to install Python with various modules and just tar it up and copy to different machines. virtualenv saved me tons of time.

This is an interesting problem that I ran into. The problem definition goes like this. You are given an array of integers. The array is considerably large (say 5 million elements). You are given two inputs: an index i in the array and an integer value v . Start searching the array for value v from the index i , and expand your search towards the two edges of the array. Return the index where the value v occurs closest to index  i . If the value v  occurs on both sides of index i at an equal distance, return the lower of the two indices. If the value v  does not occur at all, return -1. It was very interesting to solve this problem. Give it a shot, you might also like it.

Earlier I blogged about a little hurdle that I ran into when I was trying to install Emacs 24.1. Since Emacs 24.3.1 was released recently, I thought I would give it a spin. After downloading the source code , I unzipped the source code and ran configure command. I got a similar error that I was getting before. checking for libXaw... configure: error: No X toolkit could be found. If you are sure you want Emacs compiled without an X toolkit, pass --with-x-toolkit=no to configure. Otherwise, install the development libraries for the toolkit that you want to use (e.g. Gtk+) and re-run configure. I am trying to install Emacs on Mint 14. So to satisfy all the dependencies, I installed the missing packages. sudo apt-get update sudo apt-get install libgtk2.0-dev libtiff4-dev libgif-dev libpng12-dev libxpm-dev libncurses-dev libjpeg-dev libjpeg8-dev libjpeg-turbo8-dev After installing the dependencies, the usual three step process worked fine. ./configure ...