Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Obtaining a web server stack trace in Linux/Unix (2000998)

Symptoms

    Stack dumps from a debugger may be useful if you experience any of these symptoms:
    • Server crashes and creates a coredump file
    • Server hangs or is unresponsive
    • The error log contains the error message:

      possible core dump at <path>

    Purpose

    This article provides information on obtaining a stack trace against a core dump or running server process. When a server crashes or hangs, you can use the stack trace to troubleshoot the issue.
     
    Note: The stack trace is also known as a traceback, backtrace, or unwind.

    Resolution

    Caution: Before proceeding, ensure that you are aware of the following points about web server stack traces:
    • When analyzing a coredump after a server failure, it is important that you do it on the server where it occurred, so that the debugger has access to the server environment in order to resolve code references in the coredump. Do not upload web server core files to VMware Support; the stack trace is much smaller and much more helpful.
    • To analyze a coredump, you need to know the actual server executable (not the startup script). For Apache HTTP Server, it is usually httpd . For vFabric ERS, it is httpsd.worker or httpsd.prefork depending on MPM (the default is worker). For vFabric Web Server, it is httpd.prefork, httpd.worker, or httpd.event (the default is worker).
    • If your server is hung, you can take a live stack trace on running server processes. Unless you know which process is problematic, obtain a stack trace for the control process (which is the oldest one and parent to the others) and at least one child process (which actually handles requests); ideally take a stack trace for all child processes.
    • To raise a support request, VMware recommends that you capture the full output of the debugger to a file. Do not cut and paste from your terminal; instead, use tee or shell redirection as in the examples below.
    Note: In all commands that follow, substitute the following with your environment specifics:
    • /path/to/httpd – The server executable
    • /path/to/core – A coredump file that has been left by a server crash
    • PID – A server process ID that you find via the ps command.

    Using gdb

    gdb is available for free and is suitable for most Unix-like environments. It is best suited for Linux and is also available for Solaris and other platforms.
     
    To obtain a stack trace from a core file, invoke gdb using this command:

    # gdb /path/to/httpd /path/to/core | tee trace.txt

    For a live stack trace, invoke gdb using this command:

    # gdb -p PID | tee trace.txt

    In either case, wait for initialization messages to complete and display the (gdb) prompt.
     
    After the (gdb) prompt appears, run these commands:
     
    (gdb) set pagination off
    (gdb) thread apply all bt
    (gdb) quit

    A trace.txt file that contains your stack trace is created.

    Using dbx

    dbx is available on AIX or Solaris. You can invoke dbx in the same way as gdb, but the later commands are different.

    To obtain a stack trace from a core file, invoke dbx using this command:

    # dbx /path/to/httpd /path/to/core | tee trace.txt

    For a live stack trace, invoke dbx using this command:

    # dbx -a PID | tee trace.txt

    In either case, wait for initialization messages to complete and display the (dbx) prompt.
     
    After the (dbx) prompt appears:
    1. Run the info threads command to list all threads.
    2. For each thread, run the thread # command to change to the thread and run the where command to trace the thread.

    A trace.txt file that contains your stack trace is created.

    Using pstack and pflags

    pstack and pflags are standard Solaris tools.

    For a stack trace on a core dump, run these commands:

    # pstack /path/to/core  > trace.txt
    # pflags /path/to/core  >> trace.txt
     
    Note: For a core, pflags indicates where the fault occurred. Other debuggers include this with the trace.

    For a stack trace on a live process, run this command:

    # pstack PID > trace.txt

    A trace.txt file that contains your stack trace is created.

    Additional Information

    For related information, see the Apache Debugging Guide.

    Tags

    stack-trace

    Request a Product Feature

    To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

    Feedback

    • 3 Ratings

    Did this article help you?
    This article resolved my issue.
    This article did not resolve my issue.
    This article helped but additional information was required to resolve my issue.
    What can we do to improve this information? (4000 or fewer characters)
    • 3 Ratings
    Actions
    KB: