How to enable coredump and analyze the backtrace

to check the ulimit setting

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 40
file size               (blocks, -f) unlimited
pending signals                 (-i) 19553
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 19553
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

core file size is 0 means that no coredump file can be created.

check the kernel core pattern

$ sudo sysctl -a | grep kernel.core_pattern
kernel.core_pattern = %e.%p.%s.core

to enable coredump file

ulimit -c unlimited

we could add this command to ~/.bashrc

we can also set the coredump file name format like this:

sudo sysctl -w kernel.core_pattern=%e.%p.%s.core

core pattern naming

Naming of core dump files
   By default, a core dump file is named core, but the /proc/sys/kernel/core_pattern file (since  Linux  2.6  and
   2.4.21)  can  be  set  to  define a template that is used to name core dump files.  The template can contain %
   specifiers which are substituted by the following values when a core file is created:

       %%  a single % character
       %c  core file size soft resource limit of crashing process (since Linux 2.6.24)
       %d  dump mode—same as value returned by prctl(2) PR_GET_DUMPABLE (since Linux 3.7)
       %e  executable filename (without path prefix)
       %E  pathname of executable, with slashes ('/') replaced by exclamation marks ('!') (since Linux 3.0).
       %g  (numeric) real GID of dumped process
       %h  hostname (same as nodename returned by uname(2))
       %i  TID of thread that triggered core dump, as seen in the PID  namespace  in  which  the  thread  resides
           (since Linux 3.18)
       %I  TID of thread that triggered core dump, as seen in the initial PID namespace (since Linux 3.18)
       %p  PID of dumped process, as seen in the PID namespace in which the process resides
       %P  PID of dumped process, as seen in the initial PID namespace (since Linux 3.12)
       %s  number of signal causing dump
       %t  time of dump, expressed as seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC)
       %u  (numeric) real UID of dumped process

add debug info when compile app

g++ -g test.cpp -o test

analyze coredump file

gdb ./test test.7105.11.core

then type

bt

to show the backtrace info, like this

$ gdb ./test test.7105.11.core
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test/build/bin/test...done.
[New LWP 7105]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./test/build/bin/test'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055e71fafafdd in debugtrace_test_crash ()
    at /debugtrace/test/test.cpp:25
25     *ptr = 1; // will crash here
(gdb) bt
#0  0x000055e71fafafdd in debugtrace_test_crash ()
    at /debugtrace/test/test.cpp:25
#1  0x000055e71fafb01a in debugtrace_test_func12 ()
    at /debugtrace/test/test.cpp:31
#2  0x000055e71fafb04d in debugtrace_test_func11 ()
    at /debugtrace/test/test.cpp:35
#3  0x000055e71fafb080 in debugtrace_test_func10 ()
    at /debugtrace/test/test.cpp:39
#4  0x000055e71fafb0b3 in debugtrace_test_func9 ()
    at /debugtrace/test/test.cpp:43
#5  0x000055e71fafb0e6 in debugtrace_test_func8 ()
    at /debugtrace/test/test.cpp:47
#6  0x000055e71fafb119 in debugtrace_test_func7 ()
    at /debugtrace/test/test.cpp:51
#7  0x000055e71fafb14c in debugtrace_test_func6 ()
    at /debugtrace/test/test.cpp:55
#8  0x000055e71fafb17f in debugtrace_test_func5 ()
    at /debugtrace/test/test.cpp:59
#9  0x000055e71fafb1b2 in debugtrace_test_func4 ()
    at /debugtrace/test/test.cpp:63
#10 0x000055e71fafb1e5 in debugtrace_test_func3 ()
    at /debugtrace/test/test.cpp:67
#11 0x000055e71fafb218 in debugtrace_test_func2 ()
    at /debugtrace/test/test.cpp:71
#12 0x000055e71fafb24b in debugtrace_test_func1 ()
    at /debugtrace/test/test.cpp:75
#13 0x000055e71fafb3a4 in main ()
    at /debugtrace/test/test.cpp:104
(gdb) 
(gdb) q
[~/debugtrace]$ 

we can see that the Program terminated with signal SIGSEGV at debugtrace_test_crash() function.

QNX系统应用出现coredump的解析步骤

加载工具链

source qnxsdp-env.sh

使用gdb加载app和coredump文件

ntoaarch64-gdb ./app ./app.core

然后输入

bt

输入q退出gdb

示例如下

Program terminated with signal SIGBUS, Bus error.
#0  0x0000000000000001 in ?? ()
(gdb) bt
#0  0x0000000000000001 in ?? ()
#1  0x00000000004139ac in std::function<void (std::string const&, MessageHeader const&)>::function(std::function<void (std::string const&, MessageHeader const&)> const&) ()
#2  0x000000000040e694 in std::pair<std::string const, std::function<void (std::string const&, MessageHeader const&)> >::pair<char const (&) [20], std::function<void (std::string const&, MessageHeader const&)>&, true>(char const (&) [20], std::function<void (std::string const&, MessageHeader const&)>&) ()
#3  0x0000000000408d94 in Proxy::Proxy() ()
#4  0x00000000004081e0 in Proxy::getInstance() ()
#5  0x0000000000400cb8 in __static_initialization_and_destruction_0(int, int) ()
#6  0x0000000000400d04 in _GLOBAL__sub_I_main ()
#7  0x0000000000665c70 in _preinit_array (start=<optimized out>, end=<optimized out>) at /builds/workspace/710-SDP/build_aarch64/lib/c/support/_initfini.c:25
#8  0x00000000004009f4 in _start () at /builds/workspace/710-SDP/build_aarch64/lib/c/startup/aarch64/crt1.S:73
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)