Virtual IP and timeouts

Let us say you are writing a client application that connects to a server and maintains a persistent connection. Whenever there is some user intervention, for e.g. typing some input, the client application sends the user input to the server. And prints back the output produced by the server. (Sounds like telnet?)

One thing your client program doesn't know about is that the server it is connecting to makes use of a virtual IP! Consider the case when your client program initiates a connection and the connection is in ESTABLISHED state. Right in the middle of the connection, the server drops the virtual IP address and for some strange reason it never gets reassigned to another host. In that case, neither the client program nor the server program will know the event of removal of the IP address. The server application would still be listening in the removed IP address and would still be holding the ESTABLISHED connections through removed IP address. Likewise, the client would still be holding an ESTABLISHED connection to the server.

If you observe the way Linux handles writes on sockets, when the client program writes into the socket, the written bytes will be copied to the kernel buffer and the write call will immediately return. The kernel will perform a best effort delievery attempt to deliver the bytes that you have written. Remember that the packets would never be delivered to the server in case of virtual IP removal. Worst part of this is the client doesn't know about this at all.

If you look at the server part, some of the servers have an inactivity timeout after which the server will close the connection. For e.g. Apache. So the server wouldn't keep these open sockets forever.

Hmm ... Unless the client opens another connection to the server, which would timeout, there is no way the client would know that its unable to reach the server.

So the moral of the story is, if you suspect that the server might be using a virtual IP and it might be dropped and never be reassigned to another server, build both the client and the server programs with proper timeouts for read/write/inactivity. Yes, keeping a time out for inactivity is very important.

Comments

Post a Comment

Popular posts from this blog

Gotchas with DBCP

I had to make use of DB connection pool in one of the applications that I was implementing. Since I am new to Java, everyone in the team was suggesting why don't you make use of Commons DBCP. Being a fan of reuse, I decided to make use of DBCP and started reading about the parameters that are exposed so that I can customize DBCP to fit my needs. While reading through the available configuration parameters , I was thinking, "I don't need to check the connection to be bad on every borrow or return. It is sufficient if I check for being bad only when the connection is sitting idle in the pool." So I set the testWhileIdle to be true and set the validationQuery to be "SELECT 1 FROM DUAL". I was thinking that the configuration that I had was the optimal for the situation in hand. But one thing I overlooked was, the default value for the testOnBorrow is true. So the moment you set the validationQuery to be any non-null value, every connection you borrow from the po
Read more

A note on Java's Calendar set() method

Remember that the Calendar's internal fields include year, month, date, hour, minutes, seconds, milliseconds and time zone. Whenever you are calling a set() method with multiple fields, like set(year, month, date), it will not affect the rest of the fields. Remember that there is no set() method with multiple fields available to set the milliseconds. If you would like to set the milliseconds, you must use set(Calendar.MILLISECOND, value). Likewise, if you are planning to set all the fields, its a good idea to reset all the fields using clear() method. This will clear milliseconds as well. Most of the times, millisecond field may not be of interest to you. But if you are going to use the UTC milliseconds, by calling getTimeInMillis(), then make sure you set the right values for milliseconds as well.
Read more

The mysterious ORA-03111 error

Recently one of the applications that I developed started throwing exceptions, that had the following message: SQL state [72000]; error code [1013]; ORA-03111: break received on communication channel When I googled around, I couldn't come across anything useful. Sadly enough most of the sites just showed the documentation for that error, without any explanation from anyone experiencing that issues. So here you go, with the best possible explanation that I could come up with. My application sets two things on the connection that is throwing this exception: It sets the fetchSize to be 2500 rows It sets the query timeout to be 10 seconds The database server and the application are separated over a long latency network (actually there is a NetEm box that emulates the long latency between these two boxes) which has a latency characteristic of 50+/-5 milliseconds. This is the whole setup. It is important to understand how the timeout is handled by the Oracle client (in my case JDBC clien
Read more