A core dump file captures a program’s memory at the moment of a crash, providing clues to diagnose the root cause. Understanding how to read core dump files in linux is essential for any developer or system administrator who deals with application crashes. This guide will walk you through the process step by step, from enabling core dumps to analyzing them with tools like GDB.
What Is A Core Dump File In Linux
A core dump is a file that contains the memory image of a running process at the time it crashed. Think of it as a snapshot of the program’s state. When a Linux program crashes due to a segmentation fault, assertion failure, or other fatal error, the kernel can write this memory image to disk. The file is typically named “core” or “core.PID” by default.
Core dumps are invaluable for debugging because they show you the exact values of variables, the call stack, and the register states at the crash moment. Without a core dump, you often have to guess what went wrong or rely on incomplete log messages.
How To Enable Core Dumps In Linux
Before you can read core dumps, you need to make sure your system is configured to generate them. Many Linux distributions disable core dumps by default to save disk space. Here is how to enable them.
Check Current Core Dump Settings
Run the following command to see the current core dump size limit:
ulimit -c
If it returns 0, core dumps are disabled. You can enable them temporarily with:
ulimit -c unlimited
This setting only applies to your current shell session. To make it permanent, add the line to your shell’s startup file like ~/.bashrc.
Configure The Core Dump Location
By default, core dumps are written to the current working directory of the crashing process. You can change this via the /proc/sys/kernel/core_pattern file. For example:
echo "/tmp/core.%p" | sudo tee /proc/sys/kernel/core_pattern
This writes core dumps to /tmp with the process ID in the filename. To make this change permanent, edit /etc/sysctl.conf and add kernel.core_pattern=/tmp/core.%p.
How To Read Core Dump Files In Linux Using GDB
Now that you have a core dump file, you need a debugger to read it. GDB (GNU Debugger) is the standard tool for this task. Here is a step-by-step guide on how to read core dump files in linux with GDB.
Step 1: Install GDB
If GDB is not already installed, use your package manager:
sudo apt install gdb (Debian/Ubuntu) or sudo yum install gdb (RHEL/CentOS)
Step 2: Load The Core Dump
You need the executable that crashed and the core dump file. Run:
gdb /path/to/executable /path/to/core
For example:
gdb ./myprogram core.1234
GDB will load the core dump and show you the line of code where the crash occured.
Step 3: Examine The Backtrace
Once inside GDB, the first thing to do is look at the call stack. Type:
bt or backtrace
This shows you the sequence of function calls that led to the crash. The top of the stack is where the program died. If you have debug symbols (compiled with -g), you will see function names and line numbers.
Step 4: Inspect Variables And Memory
You can print variable values at the crash point. For example:
print my_variable
Or examine memory at a specific address:
x/10x 0x7fff1234
This shows 10 hexadecimal words starting at that address. Use info locals to list all local variables in the current frame.
Step 5: Switch Between Frames
If you need to see variables in a different function call, switch frames:
frame 2 (or f 2)
Then use info locals or print to inspect that frame’s context.
Common Causes Of Crashes Found In Core Dumps
When you read core dumps, you will often see patterns. Here are the most frequent culprits.
- Segmentation fault (SIGSEGV) – Accessing invalid memory, like dereferencing a NULL pointer.
- Abort (SIGABRT) – Triggered by
assert()failures orabort()calls. - Floating point exception (SIGFPE) – Division by zero or other arithmetic errors.
- Bus error (SIGBUS) – Misaligned memory access or accessing a non-existent physical address.
- Stack overflow – Infinite recursion or excessive local variable allocation.
Each signal leaves clues in the core dump. For example, a NULL pointer dereference will show address 0x0 in the instruction pointer or memory access.
How To Read Core Dump Files In Linux Without Debug Symbols
Sometimes you only have a stripped binary without debug symbols. Reading core dumps in this case is harder but not impossible. You can still get the backtrace, but function names will be missing. Instead, you see memory addresses.
To make sense of these addresses, you need the symbol table from the original build. If you have a separate debuginfo package (like myprogram-dbg), you can install it and GDB will use it. Alternatively, use objdump or nm on the executable to find function offsets.
For example:
objdump -t myprogram | grep function_name
Then match the address from the backtrace to the offset in the binary.
Using Other Tools To Analyze Core Dumps
GDB is the most powerful tool, but there are alternatives. Here are a few.
Strace And Core Dumps
Strace records system calls, but it does not read core dumps directly. However, if you have a core dump from a process that was being traced, you can correlate the system call log with the crash state.
Valgrind
Valgrind is a memory debugging tool that can detect issues before they cause crashes. It does not read core dumps, but it can prevent them. Use it to find memory leaks and invalid accesses during development.
Crash Utility
The crash utility is designed for analyzing kernel core dumps (vmcore). It is not for user-space programs, but if you are debugging the Linux kernel itself, this is the tool to use.
How To Read Core Dump Files In Linux With Python And GDB
You can automate core dump analysis using GDB’s Python scripting interface. This is useful for batch processing or custom reports. Here is a simple example.
Create a Python script like analyze_core.py:
import gdb
gdb.execute("bt")
gdb.execute("info registers")
Then run:
gdb -batch -x analyze_core.py ./executable core
This runs the commands automatically and prints the output. You can extend the script to parse specific variables or check for common crash patterns.
Best Practices For Core Dump Management
Core dumps can be large and contain sensitive data. Follow these practices to manage them effectively.
- Limit core dump size with
ulimit -c SIZEto avoid filling disk. - Use a dedicated directory like
/var/crashwith proper permissions. - Compress core dumps automatically using a cron job or systemd service.
- Never share core dumps publicly as they may contain passwords or keys.
- Delete old core dumps regularly to free space.
Troubleshooting Common Core Dump Issues
Sometimes core dumps do not generate or are incomplete. Here are solutions to common problems.
Core Dump Not Created
Check the following:
- Is
ulimit -cset to a non-zero value? - Does the process have write permission to the target directory?
- Is the filesystem full?
- Is the core pattern valid? Check
/proc/sys/kernel/core_pattern.
Core Dump Is Truncated
If the core dump file is smaller than expected, the process might have been killed before the dump completed. This can happen if the system runs out of memory or disk space during the dump. Increase the core dump size limit or free up resources.
GDB Cannot Open Core Dump
If GDB says “not a core dump” or “file format not recognized”, the file might be corrupted or not a valid ELF core file. Verify with file core – it should say “ELF 64-bit LSB core file”. If not, the dump failed.
How To Read Core Dump Files In Linux For Multithreaded Programs
Multithreaded programs produce core dumps that contain state for all threads. When you load the core dump in GDB, it shows the thread that crashed by default. To see other threads, use:
info threads
Then switch to a specific thread:
thread 2
Each thread has its own backtrace and local variables. This is useful for debugging race conditions or deadlocks. You can also examine mutexes and condition variables with info mutexes if your program uses pthreads.
Analyzing Core Dumps From Production Systems
Production core dumps often come from stripped binaries without debug symbols. To analyze them effectively, you need to match the exact build version. Always keep a copy of the debug symbols for each release. Use a symbol server or store them in a version control system.
If you have the debug symbols, you can load them separately in GDB:
gdb -s /path/to/debug/symbols -e /path/to/executable -c core
This gives you full symbolic information without needing the original build directory.
How To Read Core Dump Files In Linux Using Automated Tools
For large-scale debugging, manual analysis is time-consuming. Consider using automated tools like abrt (Automatic Bug Reporting Tool) or systemd-coredump. These tools collect core dumps and can generate reports with backtraces automatically.
Systemd-coredump is built into many modern Linux distributions. To use it, enable the service:
sudo systemctl enable systemd-coredump
Then core dumps are stored in /var/lib/systemd/coredump/. You can list them with coredumpctl list and analyze one with coredumpctl debug, which launches GDB automatically.
Frequently Asked Questions
What Is The Command To Read A Core Dump File In Linux?
The primary command is gdb /path/to/executable /path/to/core. Once inside GDB, use bt to see the backtrace and print to inspect variables.
How Do I Enable Core Dumps Permanently?
Edit /etc/security/limits.conf to set soft core unlimited for your user, and set kernel.core_pattern in /etc/sysctl.conf to specify the location.
Can I Read A Core Dump Without The Original Executable?
No, you need the exact executable that crashed. The core dump contains memory addresses that are meaningless without the binary’s symbol table.
Why Is My Core Dump File Empty Or Very Small?
This usually means the process was killed before the dump completed, or the core dump size limit is too low. Check disk space and increase the ulimit.
How Do I Extract A Backtrace From A Core Dump Automatically?
Use gdb -batch -ex "bt" -ex "quit" ./executable core to print the backtrace to stdout without interactive mode.
Conclusion
Reading core dump files in Linux is a critical skill for diagnosing program crashes. By enabling core dumps, using GDB to examine the backtrace and variables, and understanding common crash patterns, you can quickly find and fix bugs. Remember to manage core dumps carefully to avoid security risks and disk space issues. With practice, you will be able to read core dumps efficiently and reduce debugging time significantly.