My answer goes like this:
- Run the system through a debugger that would give you a fair idea about the system (where to start, what are all the functions called, etc.)
- Run the system under truss or strace (or whichever tool is applicable to your platform), which will give you a very good idea of what are all the resources the system is using. (INI files, resource files, etc)
- Observe what functions are called while different functionalities are accessed in the system (what happens in my server after I click the "Submit" button, what is the function invocation sequence when I login, etc.)
Let us take the case of a Java system. How would you find out the sequence of function calls when some functionality is invoked? Running your program under a debugger is a lot more painful. So let that not be your first resource, if not the last. We really need a digest of the function calls, rather than we stepping through the execution.
I came across this excellent tutorial by Andrew Wilcox on building our own call profiler. I believe that this is an excellent starting point. You can find more about how to write your own profiler here. Remember that you will have to slightly modify the source code present in the tutorial to make it log all the function calls, rather than just the function being profiled. That tutorial lists a decent set of references too. You might find most of them very useful, particularly the one on the JNI (Java Native Interface).
You are most welcome to share your thoughts on the ideas presented. I would be more than glad to hear them.