Process crash debug could be improved automatizing core dump analysis, especially on Linux embedded systems where on-the-fly debug is not feasible and the post-mortem analysis of core dumps is the best way to analyse process crashes stack traces.

Prepare the enviroinment

Core dumps generated in cross-compiled enviroinment analysis requires binaries and linked libraries, including debug symbols, related to crashed process. First copy binaries and libraries (or the whole filesystem) in release/ folder. Then, install in your host machine the GNU debugger (gdb) for the target architecture: in this example I use arm-none-linux-gnueabi-gdb for ARM enviroinments. Please refer to your Linux packaging system or to official documentation to get proper gdb for the architecture.

Configure GDB init

Configure GDB to find properly binaries and linked libraries. Create an init file in release/gdb-init.txt, as described, replacing $sysroot_release with your release/ absolute path.

# Set filesystem root folder
set sysroot $sysroot_release

# Prepend prepared filesystem path
set solib-absolute-prefix $sysroot_release

# If external debug symbols are used, set correct references to them
set substitute-path /usr/src/debug $sysroot_release/usr/src/debug
set debug-file-directory $sysroot_release/usr/lib/debug

# Set library path
set solib-search-path $sysroot_release/usr/lib/

The described GDB configuration is enough to analyse manually the core dump. Run interactive gdb session if you want to analyse manually cores/core.foo generated by /usr/bin/foo binary.

$ arm-none-linux-gnueabi-gdb "releases/usr/bin/foo" "cores/core.foo" -ix "releases/gdbinit.txt"

Googling around you can find further GDB initialization useful to append to init file, for example some procedures to pretty print common standard C++ objects and structures.

Configure GDB rc file

To extract debug informations in a complete automatic way, prepare an rc file in release/gdb-rc.txt with commands to execute:

# Show debug symbols availability 
echo \n== Info sharedlibrary\n
info sharedlibrary

# Show running threads
echo \n== Info threads\n
info threads

# Show backtrace summary
echo \n== Backtrace\n
backtrace

# Show full backtrace of all threads
echo \n== Thread apply all backtrace full\n
thread apply all backtrace full

# Show registers values
echo \n== Info registers\n
info registers

# Disassemble last run functions
echo \n== Disas\n
disas

In this way you can extract automatically main informations about crash, for example simply running the following command to save the output in analysis/core.foo.analysis.txt:

$ arm-none-linux-gnueabi-gdb "releases/usr/bin/foo" "cores/core.foo" -ix "releases/gdbinit.txt" -x "releases/gdbrc.txt" -batch > "analysis/core.foo.analysis.txt"

Have a good core crunching.