Roy's musings

Designing a distributed system is hard enough. Even harder to design a distributed system that is reliable. There are many best practices that you can follow to make a reliable distributed system. Based on an issue that I recently troublehooted, there are a couple of them that I think are critical: Enabling TCP keep-alive between the processes if you are using TCP Performing all IO operations with a time out My advise is based on my experience in Linux. But I think it should be applicable to other operating systems as well. When a host goes down with a kernel panic, none of the established connections are closed by sending a FIN or RESET packet. This cauases trouble that the peer process doesn't know about the other end of the communication being gone. When you enable TCP keep-alive, the kernel sends a zero length packet as per the configuration. Hence if the peer has died due to kernel panic, the zero length packet will not be ACKed. Thus, the peer death is detected. If th...

I was troubleshooting an issue with one of the clients. And I desperately needed them to run the netcat tool and send me the output. But their OS is windows. So I was looking for the netcat tool for windows. Here it is: http://joncraton.org/files/nc111nt.zip

Recently one of the applications that I developed started throwing exceptions, that had the following message: SQL state [72000]; error code [1013]; ORA-03111: break received on communication channel When I googled around, I couldn't come across anything useful. Sadly enough most of the sites just showed the documentation for that error, without any explanation from anyone experiencing that issues. So here you go, with the best possible explanation that I could come up with. My application sets two things on the connection that is throwing this exception: It sets the fetchSize to be 2500 rows It sets the query timeout to be 10 seconds The database server and the application are separated over a long latency network (actually there is a NetEm box that emulates the long latency between these two boxes) which has a latency characteristic of 50+/-5 milliseconds. This is the whole setup. It is important to understand how the timeout is handled by the Oracle client (in my case JDBC clien...

I was trying to install the FreeBSD 7.0 in my VMWare virtual machine (VM). After I downloaded the installer disk's ISO image, I connected that ISO file as my CDROM drive in my VM and booted my VM. I should mention that my VM is in NAT mode, as I had decided to install the FreeBSD over the network. When the installation tries to download the distro files from the FTP servers, it just kept saying it could not connect to the FTP server. I wasted around an hour or so without thinking about the basics (Damn!) that I am behind a NAT and I am trying to download files using active FTP mode. Then I tried to connect in passive mode and everything worked fine. I don't want to repeat what many people have already clearly explained about passive mode of FTP. Here is an excellent tutorial that talks about active and passive modes of FTP. Moral of the story is, don't forget that active FTP doesn't work (generally :-) if you are behind a NAT. I would like to add only one thing: I thin...

Whenever you want to obtain a dynamic IP address for your Linux/Unix machine from a DHCP server, you should use dhclient utility. Both DHCP request and responses are UDP requests. If you use a sniffer to identify the pattern of the request responses, the following is what you might see. The following was taken from dhclient running in FreeBSD 6.1. Len SrcIP SrcMACAddr DestIP DestMACAddr Protocol 342 0.0.0.0 00:0c:29:c1:13:81 255.255.255.255 ff:ff:ff:ff:ff:ff UDP 62 192.168.49.254 00:50:56:ef:75:a8 192.168.49.128 ff:ff:ff:ff:ff:ff ICMP ping request 342 192.168.49.254 00:50:56:ef:75:a8 192.168.49.128 00:0c:29:c1:13:81 UDP 60 192.168.49.128 00:0c:29:c1:13:81 ff:ff:ff:ff:ff:ff ARP request If you observe the source and destination (IP, MACAddr) patterns, it is easy to appreciate what happens. Here is what happens: A DHCP request is sent on the network. Both destination MACAddr and IPAddr are broadcast addresses. The DHCP server chooses and IP ...