The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
Search the VMware Knowledge Base (KB)
View by Article ID
Obtaining a web server stack trace in Linux/Unix (2000998)
Symptoms
Stack dumps from a debugger may be useful if you experience any of these symptoms:
Server crashes and creates a coredump file
Server hangs or is unresponsive
The error log contains the error message:
possible core dump at <path>
Purpose
This article provides information on obtaining a stack trace against a core dump or running server process. When a server crashes or hangs, you can use the stack trace to troubleshoot the issue.
Note: The stack trace is also known as a traceback, backtrace, or unwind.
Resolution
Caution: Before proceeding, ensure that you are aware of the following points about web server stack traces:
When analyzing a coredump after a server failure, it is important that you do it on the server where it occurred, so that the debugger has access to the server environment in order to resolve code references in the coredump. Do not upload web server core files to VMware Support; the stack trace is much smaller and much more helpful.
To analyze a coredump, you need to know the actual server executable (not the startup script). For Apache HTTP Server, it is usually httpd . For vFabric ERS, it is httpsd.worker or httpsd.prefork depending on MPM (the default is worker). For vFabric Web Server, it is httpd.prefork, httpd.worker, or httpd.event (the default is worker).
If your server is hung, you can take a live stack trace on running server processes. Unless you know which process is problematic, obtain a stack trace for the control process (which is the oldest one and parent to the others) and at least one child process (which actually handles requests); ideally take a stack trace for all child processes.
To raise a support request, VMware recommends that you capture the full output of the debugger to a file. Do not cut and paste from your terminal; instead, use tee or shell redirection as in the examples below.
Note: In all commands that follow, substitute the following with your environment specifics:
/path/to/httpd – The server executable
/path/to/core – A coredump file that has been left by a server crash
PID – A server process ID that you find via the ps command.
Using gdb
gdb is available for free and is suitable for most Unix-like environments. It is best suited for Linux and is also available for Solaris and other platforms.
To obtain a stack trace from a core file, invoke gdb using this command:
# gdb /path/to/httpd /path/to/core | tee trace.txt
For a live stack trace, invoke gdb using this command:
# gdb -p PID | tee trace.txt
In either case, wait for initialization messages to complete and display the (gdb) prompt.
After the (gdb) prompt appears, run these commands:
(gdb) set pagination off
(gdb) thread apply all bt
(gdb) quit
A trace.txt file that contains your stack trace is created.
Using dbx
dbx is available on AIX or Solaris. You can invoke dbx in the same way as gdb, but the later commands are different.
To obtain a stack trace from a core file, invoke dbx using this command:
# dbx /path/to/httpd /path/to/core | tee trace.txt
For a live stack trace, invoke dbx using this command:
# dbx -a PID | tee trace.txt
In either case, wait for initialization messages to complete and display the (dbx) prompt.
After the (dbx) prompt appears:
Run the info threads command to list all threads.
For each thread, run the thread # command to change to the thread and run the where command to trace the thread.
A trace.txt file that contains your stack trace is created.
Using pstack and pflags
pstack and pflags are standard Solaris tools.
For a stack trace on a core dump, run these commands:
