Process crash debug could be improved automatizing core dump analysis, especially on Linux embedded systems where on-the-fly debug is not feasible and the post-mortem analysis of core dumps is the best way to analyse process crashes stack traces.
Core dumps generated in cross-compiled enviroinment analysis requires binaries and linked libraries, including debug symbols, related to crashed process. First copy binaries and libraries (or the whole filesystem) in
release/ folder. Then, install in your host machine the GNU debugger (gdb) for the target architecture: in this example I use
arm-none-linux-gnueabi-gdb for ARM enviroinments. Please refer to your Linux packaging system or to official documentation to get proper gdb for the architecture.
Configure GDB to find properly binaries and linked libraries. Create an init file in
release/gdb-init.txt, as described, replacing
$sysroot_release with your
release/ absolute path.
# Set filesystem root folder set sysroot $sysroot_release # Prepend prepared filesystem path set solib-absolute-prefix $sysroot_release # If external debug symbols are used, set correct references to them set substitute-path /usr/src/debug $sysroot_release/usr/src/debug set debug-file-directory $sysroot_release/usr/lib/debug # Set library path set solib-search-path $sysroot_release/usr/lib/
The described GDB configuration is enough to analyse manually the core dump. Run interactive gdb session if you want to analyse manually
cores/core.foo generated by
$ arm-none-linux-gnueabi-gdb "releases/usr/bin/foo" "cores/core.foo" -ix "releases/gdbinit.txt"
Googling around you can find further GDB initialization useful to append to init file, for example some procedures to pretty print common standard C++ objects and structures.
To extract debug informations in a complete automatic way, prepare an rc file in
release/gdb-rc.txt with commands to execute:
# Show debug symbols availability echo \n== Info sharedlibrary\n info sharedlibrary # Show running threads echo \n== Info threads\n info threads # Show backtrace summary echo \n== Backtrace\n backtrace # Show full backtrace of all threads echo \n== Thread apply all backtrace full\n thread apply all backtrace full # Show registers values echo \n== Info registers\n info registers # Disassemble last run functions echo \n== Disas\n disas
In this way you can extract automatically main informations about crash, for example simply running the following command to save the output in
$ arm-none-linux-gnueabi-gdb "releases/usr/bin/foo" "cores/core.foo" -ix "releases/gdbinit.txt" -x "releases/gdbrc.txt" -batch > "analysis/core.foo.analysis.txt"
Have a good core crunching.