Where do I find core dump files, and how do I view and analyze the backtrace (stack trace) in one?

Question

When I run my C program on Ubuntu 20.04, I get this run-time error:

Segmentation fault (core dumped)

I really need to find and view the core file, but I can't find it anywhere. Where is it, and how do I view the backtrace in it?

Answer 1

Tested in Ubuntu 20.04.

1. Enable core files

First off, run ulimit -c to see what the max allowed size is for core files on your system. On Ubuntu 20.04 for me, mine returns 0, which means no core file can be created.

ulimit --help shows the meaning of -c:

-c  the maximum size of core files created

So, set the allowed core file size to unlimited, as shown below. Note that I think this only applies to the one terminal you run this in, and I do not think it's persistent across reboots, so you have to run this each time you want core files to be created, and in each terminal you want it to work in:

# set max core dump file size to unlimited
ulimit -c unlimited
# verify it is now set to "unlimited"
ulimit -c

That's it! Now, run your program and if it crashes and does a "core dump" it will dump the core as a core file into the same directory you were in when you called the executable. The name of the file is simply "core".

UPDATE: wait, where are the core files again?

I tested all of my code and examples here, and they work for me. However, if you do not get a core file locally as described above, apparently you are not alone. So, try the following:

1. (As of Ubuntu 21.10 and 22.04): Look inside /var/lib/apport/coredump, as explained in this other answer by @guyr here.
2. Look for answers here:
   1. Where do I find the core dump in ubuntu 16.04LTS?. Possibilities (or commands to run) include:
      1. /var/crash/
      2. cat /var/log/apport.log
   2. [Lots of votes on this one!] Stack Overflow: Core dumped, but core file is not in the current directory?
      1. systemd may be causing some problems here
3. If your core dump files don't work like mine, leave a comment with what version of Linux you have (ex: Ubuntu 22.04), what kernel version you have (run cat /proc/version), and where your core dump files are.
   1. Here are my own answers: on Linux Ubuntu 18.04 and 20.04, ulimit -c unlimited causes core dump files to appear right in the dir where I am. I have to manually delete old ones for new ones to be created. On my Ubuntu 18.04 machine, cat /proc/version shows Linux version 4.15.0-194-generic (buildd@lcy02-amd64-052) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)).

2. View the backtrace in gdb

You should have built your C or C++ program with debug symbols on, in order to see useful information in your core file. Without debug symbols, you can only see the addresses of the functions called, not the actual names or line numbers.

In gcc, use -ggdb -O0 to turn on debug symbols optimized for the gdb GNU debugger. You can also use -g -O0, -g3 -O0, etc, but -ggdb -O0 is best. Do we really need optimization level 0 (-O0) for this? Yes, yes we do. See my answer here: Stack Overflow: What's the difference between a compiler's -O0 option and -Og option?

Example build and run commands in C and C++: so, your full build and run commands in C or C++ might look like this:

# C build and run command for "hello_world.c"
gcc -Wall -Wextra -Werror -ggdb -O0 -std=c11 -o hello_world hello_world.c \
&& ./hello_world

# C++ build and run command for "hello_world.c"
g++ -Wall -Wextra -Werror -ggdb -O0 -std=c++17 -o hello_world hello_world.c \
&& ./hello_world

Open the core file in gdb like this:

gdb path/to/my/executable path/to/core

Assuming you just ran path/to/my/executable, then the core file will be in the same directory you were just in when the core was dumped, so you can just run this:

gdb path/to/my/executable core

In gdb, view the backtrace (function call stack at the time of the crash) with:

bt
# or (exact same command)
where

# OR (for even more details, such as seeing all arguments to the functions--
# thanks to Peter Cordes in the comments below)
bt full

# For gdb help and details, see:
help bt
# or
help where

IMPORTANT: when a core dump occurs, it does NOT automatically overwrite any pre-existing core file in your current directory with a new one, so you must manually remove the old core file with rm core PRIOR TO generating the new core file when your program crashes, in order to always have the latest core file to analyze.

3. Try it out

1. In a terminal, run sleep 30 to start a process sleeping for 30 seconds.
2. While it is running, press Ctrl + \ to force a core dump. You'll now see a core file in the directory you are in.
3. Since we don't have an executable for this with debugging symbols in it, we will just open up the core file in gdb instead of the executable file with symbols + the core file. So, run gdb -c core to open the core file just created by the forced crash.
4. You'll see this. Notice it knows what command you called (sleep 30) when the core dump occurred:

   Core was generated by `sleep 30'.
   Program terminated with signal SIGQUIT, Quit.
   #0  0x00007f93ed32d334 in ?? ()
   (gdb) 
   

5. Run bt or where to see the backtrace. You'll see this:

   (gdb) bt
   #0  0x00007f93ed32d334 in ?? ()
   #1  0x000000000000000a in ?? ()
   #2  0x00007f93ed2960a5 in ?? ()
   #3  0x0000000000000000 in ?? ()
   (gdb)
   

6. Those are the addresses to the functions called on the call stack. If you had debugging symbols on, you'd see a lot more info, including function names and line numbers, like this (pulled from a C program of mine):

   #10 0x00007fc1152b8ebf in __printf (format=<optimized out>) at printf.c:33
   #11 0x0000562bca17b3eb in fast_malloc (num_bytes=1024) at src/fast_malloc.c:225
   #12 0x0000562bca17bb66 in malloc (num_bytes=1024) at src/fast_malloc.c:496
   

4. Forget about core files and just run the program to the crash point in gdb directly!

As @Peter Cordes states in the comments below, you can also just run the program inside gdb directly, letting it crash there, so you have no need to open up a core file after-the-fact! He stated:

Those GDB commands are not specific to core files, they work any time you're stopped at a breakpoint. If you have a reproducible crash, it's often easier / better to run your program under GDB (like gdb ./a.out) so GDB will have the process in memory instead of a core file. The main advantage is that you can set a breakpoint or watchpoint somewhere before the thing that crashed, and single-step to see what's happening. Or with GDB's record facilities, you may be able to step backwards and see what led up to the crash, but that can be flaky, slow, and memory-intensive.

As stated above, you should have compiled your program with debugging symbols on and with Optimization Level 0, using -ggdb -O0. See the full example build and run commands in C and C++ above.

Now run the program in gdb:

# Open the executable in gdb
gdb path/to/my/executable
# Run it (if it's still crashing, you'll see it crash)
r 
# View the backtrace (call stack)
bt  
# Quit when done 
q

And if you ever need to manually log the backtrace to a log file to analyze later, you can do so like this (adapted from notes in my eRCaGuy_dotfiles repo here):

set logging file gdb_log.txt
set logging on
set trace-commands on
show logging     # prove logging is on
flush
set pretty print on
bt               # view the backtrace
set logging off  
show logging     # prove logging is back off

Done! You've now saved the gdb backtrace in file "gdb_log.txt".

References:

1. [the answer I needed is in this question itself] https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-the-current-directory
2. https://stackoverflow.com/questions/5115613/core-dump-file-analysis
3. https://stackoverflow.com/questions/8305866/how-do-i-analyze-a-programs-core-dump-file-with-gdb-when-it-has-command-line-pa/30524347#30524347
4. [very useful info, incl. the Ctrl + \ trick to force a core dump!] https://unix.stackexchange.com/questions/277331/segmentation-fault-core-dumped-to-where-what-is-it-and-why/409776#409776
5. [referenced by the answer above] https://unix.stackexchange.com/questions/179998/where-to-search-for-the-core-file-generated-by-the-crash-of-a-linux-application/180004#180004
6. [answer is in the question itself] Where do I find the core dump in ubuntu 16.04LTS?
7. [my answer] Stack Overflow: What's the difference between a compiler's -O0 option and -Og option?

Additional reading to do

1. [I STILL NEED TO STUDY & TRY THIS] How to use LD_PRELOAD with gdb: https://stackoverflow.com/questions/10448254/how-to-use-gdb-with-ld-preload

Answer 2

Found via search. I'm running Ubuntu Mate 21.10. For those running late model Ubuntu, apport will generate dumps in /var/lib/apport/coredump.

If you can't find your core dump file, cat /var/log/apport.log. When I did that, I saw:

executable does not belong to a package, ignoring
called for pid 5545, signal 11, core limit 0, dump mode 1

Notice the core limit 0, that means no core dump file will be generated. So, I ran the command shown in this post (ulimit -c unlimited), and this time apport.log showed this:

writing core dump to core._my_prog.1000.e43b2f33-4708-438c-a7d7-05062f381382.5650.795448 (limit: -1)

I couldn't find this in the current directory or the directory containing the executable, so I did a find on the entire system and found it in /var/lib/apport/coredump.

Answer 3

The location of the core file depends on /proc/sys/kernel/core_pattern

E.g. on Ubuntu 22.04:

cat /proc/sys/kernel/core_pattern

gives:

|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E

The magic leading pipe syntax means that the Linux kernel will call the program /usr/share/apport/apport with a bunch of informational arguments and then the apport executable will then handle and store the dump somewhere.

apport in particular is part of Ubuntu's ultra automated error statistics setup, I think Canonical keeps a database of all crashes by users who agree to provide them to help prioritize development https://wiki.ubuntu.com/Apport

If you want to get a simple raw core file in the current directory as Linux intended, you have to instead set it to something like core with:

echo 'core' | sudo tee /proc/sys/kernel/core_pattern

How to change it persistently across reboot: How to permanently edit the core_pattern file?

If you do that and then:

ulimit -c unlimited

the current kernel 5.15.0 dumps files with form:

core.<pid>

e.g.:

core.494536

under the current working directory.

It is also apparently possible to turn off apport with: How do I enable or disable Apport?

sudo systemctl disable apport.service

You can test this e.g. with a minimal C program:

segfault.c

#include <stdlib.h>

int main(void) {
    *(int *)0 = 1;
    return EXIT_SUCCESS;
}

compile and run with:

gcc -ggdb3 -O0 -pedantic-errors -std=c89 -Wall -Wextra -o segfault.out segfault.c
./segfault.out

How to obtain apport core dumps?

Explained at: https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-the-current-directory/47481884#47481884

Let's learn to use it if you decide to keep Ubuntu's mega opinionated apport system.

Maybe it is the for the best that non programs won't get random core.1234 files randomly created in their directories from time to time.

apport stores core dumps under .crash files under /var/crash/. .crash files are wrappers that contain the core dumps and some more system logs to help Ubuntu devs debug.

If you just run:

./segfault.out

you don't get anything. "Great" default behavior!

But Where do I find the core dump in ubuntu 16.04LTS? explains that we can look for apport logs under:

cat /var/log/apport.log

which now contains entries of type:

ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: called for pid 503173, signal 11, core limit 18446744073709551615, dump mode 1
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: ignoring implausibly big core limit, treating as unlimited
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: executable: /home/ciro/segfault.out (command line "./segfault.out")
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: executable does not belong to a package, ignoring
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: writing core dump to core._home_ciro_segfault_out.1000.57a7653e-d57e-4871-ad0d-f7b8d9f5b3b9.503173.5035472 (limit: -1)

and no file is generated as per the ignore.

https://stackoverflow.com/questions/14204961/how-to-change-apport-default-behaviour-for-non-packaged-application-crashes asks how to enable the cores so we try:

mkdir -p ~/.config/apport
printf '[main]
unpackaged=true
' >> ~/.config/apport/settings

and now it works, we have a .crash file under /var/crash:

-rw-r--r-- 1 ciro     whoopsie    0 Nov 26 22:09 _home_ciro_segfault.out.1000.upload
-rw-r----- 1 ciro     whoopsie 137K Nov 26 22:09 _home_ciro_segfault.out.1000.crash
-rw------- 1 whoopsie whoopsie   37 Nov 26 22:09 _home_ciro_segfault.out.1000.uploaded

We can then extract the core dump with:

apport-unpack /var/crash/_home_ciro_segfault.out.1000.crash segfault

which creates a directory named segfault/ containing a bunch of files split out of the .crash file, including our core dump segfault/CoreDump.

The program does crash however due to a bug, it is pitiful:

Traceback (most recent call last):
  File "/usr/bin/apport-unpack", line 77, in <module>
    pr.extract_keys(f, bin_keys, dir)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 269, in extract_keys
    raise ValueError('%s has no binary content' %
ValueError: ['separator'] has no binary content

Report at: https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1889443 But it crashes after generating our CoreDumpso we can successfully run:

gdb segfault.out segfault/CoreDump

There is also a potentially more automated way with:

apport-retrace /var/crash/_home_ciro_segfault.out.1000.crash

but it fails for non packaged programs with:

ERROR: report file does not contain one of the required fields: Package

Tested on Ubuntu 22.04.

How to prevent "this executable already crashed 2 times, ignoring"?

If you keep testing with our minimal ./segfault.out, you will notice that at some point .crash files stop being generated.

cat /var/log/apport.log

then clarifies that:

this executable already crashed 2 times, ignoring

OMG this system was not made with devs in mind!

Looking at at source code from apt source apport does not seem configurable at first sight:

    crash_counter = 0

    # Create crash report file descriptor for writing the report into
    # report_dir
    try:
        report = '%s/%s.%i.crash' % (apport.fileutils.report_dir, info['ExecutablePath'].replace('/', '_'), pidstat.st_uid)
        if os.path.exists(report):
            if apport.fileutils.seen_report(report):
                # do not flood the logs and the user with repeated crashes
                # and make sure the file isn't a FIFO or symlink
                fd = os.open(report, os.O_NOFOLLOW | os.O_RDONLY | os.O_NONBLOCK)
                st = os.fstat(fd)
                if stat.S_ISREG(st.st_mode):
                    with os.fdopen(fd, 'rb') as f:
                        crash_counter = apport.fileutils.get_recent_crashes(f)
                    crash_counter += 1
                    if crash_counter > 1:
                        write_user_coredump(
                            pid, process_start, core_ulimit, coredump_fd
                        )
                        error_log('this executable already crashed %i times, ignoring' % crash_counter)
                        sys.exit(0)
                # remove the old file, so that we can create the new one with
                # os.O_CREAT|os.O_EXCL
                os.unlink(report)

How to prevent apport from uploading/requesting to upload reports for non packaged binaries?

On Ubuntu 22.04 under:

• Settings
• Privacy
• Diagnostics
• Send error reports to Canonical

there are three possible options:

• Never
• Automatic, which seems to mean more precisely: "Always without asking"
• Manual, which in plain English means "Ask before sending" with a popup

I don't see how to prevent upload with the "Automatic" method. And "Manual" will drive you mad with popups. So the only reasonable options are "Never" or disabling apport:

• What is the 'whoopsie' process and how can I remove it?

How to analyze core dumps?

This is very Ubuntu independent, so just go for:

• https://stackoverflow.com/questions/8305866/how-do-i-analyze-a-programs-core-dump-file-with-gdb-when-it-has-command-line-pa
• https://unix.stackexchange.com/questions/89933/how-to-view-core-files-for-debugging-purposes-in-linux

Mozilla rr reverse debugging as the ultimate "core file"

Core files allow you to inspect the stack at break.

But in general what you really need to do is to go back in time to further decide the root failure cause.

The amazing Mozilla rr allows you to do that, at the cost of a larger trace file, and a slight performance hit.

Example at: https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242