Strace and defunct processes

We have a web application running in Tomcat. Today I ran into an issue that the web application started throwing some errors during start up. I wanted to figure what the application reads during the start up. So I started the whole Tomcat under strace, and started to analyze the strace output. Once I figured what was happening, I tried to stop strace with Ctrl-C, but it won't stop. So, I killed it with "kill -9". The strace is dead, but the Java process that was being traced went into a defunct state. This is what I was seeing when I did a "ps aux":
9929 1932 0.0 0.0 0 0 pts/0 Z 16:53 0:03 [java <defunct>]
Hmm ... Not good! I tried to kill this process with "kill -9", but it won't work. If I try to start another instance of Tomcat, it complains that the port 8989, where it was supposed to listen, was already in use. But if I try to do a "netstat -pan", I couldn't find any process listening in that port either. If I try to do a "telnet localhost 8989", I am able to connect. If I try to do a "/usr/sbin/lsof -n -P | grep 8989", I didn't see any process listening in that port! Wierd!

I tried to Google about defunct processes, but didn't find anything that would work.

Upon further investigation, the issue became clear. When I did a "ps auxm", I realized that there were other threds under this PID. And all those threads were under "Tracing" mode. See below (I have modified the output to fit the screen):
9929 1937 0.0 2.3 594924 48764 pts/0 T 16:53 0:00 java ...
... (10 such threads) ...
9929 1932 0.0 0.0 0 0 pts/0 Z 16:53 0:03 [java <defunct>]
The most important column is the "T" column. So, when the strace was killed, it left the process being traced and all the threads under that in being "Traced" state. I observed that any process that is defunct and being "Traced" cannot be killed with "kill -9". So the only way we can bring the process out of this state is by sending a SIGCONT signal. So I tried "kill -CONT". Voila! All those threads were reaped by init process in second.

So next time, if you see a process that is defunct, check if it is having any threads that might be in "T" state. If yes, try to send SIGCONT signal to one of the threads that is in T state. It should bring the process out of that state and the process will immediately be reaped by the init.

This issue happened in kernel version 2.4.21. I don't know if it has been addressed in later versions of kernel.

Comments

Post a Comment

Popular posts from this blog

The mysterious ORA-03111 error

Recently one of the applications that I developed started throwing exceptions, that had the following message: SQL state [72000]; error code [1013]; ORA-03111: break received on communication channel When I googled around, I couldn't come across anything useful. Sadly enough most of the sites just showed the documentation for that error, without any explanation from anyone experiencing that issues. So here you go, with the best possible explanation that I could come up with. My application sets two things on the connection that is throwing this exception: It sets the fetchSize to be 2500 rows It sets the query timeout to be 10 seconds The database server and the application are separated over a long latency network (actually there is a NetEm box that emulates the long latency between these two boxes) which has a latency characteristic of 50+/-5 milliseconds. This is the whole setup. It is important to understand how the timeout is handled by the Oracle client (in my case JDBC clien...
Read more

Note on allocationSize parameter of @SequenceGenerator while using JPA

I ran into what I thought as an issue while I was using the sequence ID generation strategy in JPA. The JPA provider I am using is Hibernate. I think sharing my experience will save someone some time. To use a sequence (For e.g. Oracle sequence) to generate values and assign them to the ID field of your persistence object, you will following something like this: @Id @Column(name = "ITEM_ID") @GeneratedValue(strategy = GenerationType.SEQUENCE, generator="ItemIdSeqGenerator") @SequenceGenerator(name="ItemIdSeqGenerator", sequenceName="ITEM_ID_SEQ", allocationSize=1) private long itemId; This means the following things: The @Id annotation says that the field itemId is a primary key. The @Column annotation says that the corresponding column in the database is ITEM_ID. The @GeneratedValue says that the value that needs to be populated in the itemId should be generated, while that object is persisted. The strategy to generate the value is to...
Read more

xVM's vboxmanage.exe command

You can view and edit the configuration of your guest OSs using the vboxmanage command. To view the list of guest OSs you have, give the following command: C:\Program Files\Sun\xVM VirtualBox>VBoxManage.exe list vms VirtualBox Command Line Management Interface Version 2.2.2 (C) 2005-2009 Sun Microsystems, Inc. All rights reserved. "MySolaris" {dc6dc85f-5583-4a1b-bf3e-969941a2cd91} "Ubuntu" {eb973bbf-d86e-4579-85eb-6ea2cd12bf95} "Debian" {3e784597-89d8-4f17-90cb-63e866c651a3} "openSUSE" {6862884e-60e1-4c65-8aab-b57ec38a3922} "Debian-Lenny" {4b561432-abac-4a27-a501-b42af956b96b} To view the specific guest, you can use the UUID of the guest. For e.g. C:\Program Files\Sun\xVM VirtualBox>VBoxManage.exe showvminfo eb973bbf-d86e-4579-85eb-6ea2cd12bf95 VirtualBox Command Line Management Interface Version 2.2.2 (C) 2005-2009 Sun Microsystems, Inc. All rights reserved. Name: Ubuntu Guest OS: Ubuntu UUID: eb973bb...
Read more