Service health checks - the right way to build them

Service health checks are ubiquitous. If you have built any software that relies upon any upstream service, I am sure you would have used some form of health check. Your software could be a stand alone program or it could be a proxy. You may also have built and exposed a service that is used as an upstream service by another software.

The worst way to perform health check is to pull a resource (like GET /hc.html, if your service is exposed as a HTTP service) or perform a TCP connect check in the same port where data is served. These are checks I call as in-band health checks. Please don't do these.

Based on my experience, the best way to expose health checks is by using an out-of-band mechanism. This means that you expose another port for performing health check. The client can perform a HTTP check or TCP check in the health check port. As a convenience, if you expose health check as HTTP service in health check port, you can consolidate multiple health checks like: GET /healthcheck/port1, GET /healthcheck/port2, etc.

One convention that could be used is to use a fixed offset port as control port for any data port. For instance, 80's control port is 81 and 443's control port is 444. Or 80's control port is 8080, and 443's control port is 8443. So when you need to decide if the service is up/down, you can check the control port. 

Now let me explain what is the benefit of the out-of-band health checks. If you would like to take the service down gracefully for maintenance, all that you have to do is to fail health checks for that port (GET /healthcheck/port1 returns a 4xx) or stop listening in the health check port. This will not impact any of the on-going requests. You can gracefully close the connections after serving the requests or after a period of idle time. This will enable you to perform scheduled maintenance like a champ without impacting any client traffic at all. Having zero impact to any client traffic should be your number one goal, when performing scheduled maintenance.

Comments

slot joker asia link alternatif said…
This comment has been removed by a blog administrator.
4/29/2022 01:05:00 AM
Post a Comment

Popular posts from this blog

The mysterious ORA-03111 error

Recently one of the applications that I developed started throwing exceptions, that had the following message: SQL state [72000]; error code [1013]; ORA-03111: break received on communication channel When I googled around, I couldn't come across anything useful. Sadly enough most of the sites just showed the documentation for that error, without any explanation from anyone experiencing that issues. So here you go, with the best possible explanation that I could come up with. My application sets two things on the connection that is throwing this exception: It sets the fetchSize to be 2500 rows It sets the query timeout to be 10 seconds The database server and the application are separated over a long latency network (actually there is a NetEm box that emulates the long latency between these two boxes) which has a latency characteristic of 50+/-5 milliseconds. This is the whole setup. It is important to understand how the timeout is handled by the Oracle client (in my case JDBC clien...
Read more

Note on allocationSize parameter of @SequenceGenerator while using JPA

I ran into what I thought as an issue while I was using the sequence ID generation strategy in JPA. The JPA provider I am using is Hibernate. I think sharing my experience will save someone some time. To use a sequence (For e.g. Oracle sequence) to generate values and assign them to the ID field of your persistence object, you will following something like this: @Id @Column(name = "ITEM_ID") @GeneratedValue(strategy = GenerationType.SEQUENCE, generator="ItemIdSeqGenerator") @SequenceGenerator(name="ItemIdSeqGenerator", sequenceName="ITEM_ID_SEQ", allocationSize=1) private long itemId; This means the following things: The @Id annotation says that the field itemId is a primary key. The @Column annotation says that the corresponding column in the database is ITEM_ID. The @GeneratedValue says that the value that needs to be populated in the itemId should be generated, while that object is persisted. The strategy to generate the value is to...
Read more

xVM's vboxmanage.exe command

You can view and edit the configuration of your guest OSs using the vboxmanage command. To view the list of guest OSs you have, give the following command: C:\Program Files\Sun\xVM VirtualBox>VBoxManage.exe list vms VirtualBox Command Line Management Interface Version 2.2.2 (C) 2005-2009 Sun Microsystems, Inc. All rights reserved. "MySolaris" {dc6dc85f-5583-4a1b-bf3e-969941a2cd91} "Ubuntu" {eb973bbf-d86e-4579-85eb-6ea2cd12bf95} "Debian" {3e784597-89d8-4f17-90cb-63e866c651a3} "openSUSE" {6862884e-60e1-4c65-8aab-b57ec38a3922} "Debian-Lenny" {4b561432-abac-4a27-a501-b42af956b96b} To view the specific guest, you can use the UUID of the guest. For e.g. C:\Program Files\Sun\xVM VirtualBox>VBoxManage.exe showvminfo eb973bbf-d86e-4579-85eb-6ea2cd12bf95 VirtualBox Command Line Management Interface Version 2.2.2 (C) 2005-2009 Sun Microsystems, Inc. All rights reserved. Name: Ubuntu Guest OS: Ubuntu UUID: eb973bb...
Read more