Analyzing Core Dumps and Debugging Techniques with GDB

Program core dumps are generated when an application crashes and cannot continue running. A core‑dump file contains a snapshot of the process state, including memory, CPU registers, program counter, and stack pointer. By examining this file, developers can locate the root cause of a crash.

The article classifies core causes into three categories: hardware/machine issues, resource exhaustion, and program bugs. A table (shown in the original article) lists common reasons for each category.

Function Stack Overview

When a core file is opened, the first thing to look at is the function call stack at the moment of the crash. The article briefly explains the function stack and the role of registers on a 64‑bit x86‑64 system.

Key registers include:

%rsp – stack top pointer

%rbp – stack base pointer

%rdi, %rsi, %rdx, %rcx, %r8, %r9 – first six function arguments

Understanding which registers are saved (Callee‑Save) is essential for stack analysis.

Function Call Mechanics

During a function call, arguments are pushed onto the stack in reverse order, unless they can be passed via registers. After arguments, the return address is pushed, which points to the instruction following the call. The article provides a diagram of the stack layout and several illustrative examples.

Using GDB to Locate Core Issues

The article outlines a step‑by‑step workflow for debugging core files with GDB:

Identify the core file location (e.g., check /proc/sys/kernel/core_pattern ).

Print variables using GDB commands such as print and x . The supported formats are listed below:

print [Expression]
print $[Previous value number]
print {[Type]}[Address]
print [First element]@[Element count]
print /[Format] [Expression]

Format options:
o – octal
x – hexadecimal
u – unsigned decimal
t – binary
f – floating point
a – address
c – character
s – string

For container types (e.g., std::string ), the article demonstrates how to inspect the internal representation:

# Convert _M_p to _Rep pointer and offset by one struct size
p *((std::string::_Rep*)(s._M_dataplus._M_p) - 1)

To dump memory to a file, use the dump command:

dump binary memory file1 s._M_dataplus._M_p s._M_dataplus._M_p+length

Locating the Faulty Code Line

When the stack trace does not directly reveal the problematic line, the article suggests using the program counter ( $rip ) together with addr2line to map the address back to source code:

# Switch to frame 20
frame 20
# Show instruction at program counter
display /i $rip
# Convert address to source line
shell /opt/compiler/gcc-8.2/bin/addr2line -e bin address

If the binary was compiled with optimizations, the article recommends rebuilding with -O0 for clearer debugging information.

Stack Repair and Irregular Core Stacks

In cases where the stack appears corrupted (e.g., many "??" entries), the article explains how to manually reconstruct the stack using the known layout of %rbp and %rsp :

# Show top of current frame and return address
x /2ag $rbp   # two 8‑byte units, a = hex, g = 8‑byte
# Recursively examine previous frames
x /2ag address

Irregular core stacks often stem from heap corruption. An example demonstrates how overwriting a std::string object's internal pointer leads to a crash during destruction.

Detecting Optimized‑Away Variables

When variables are optimized out (e.g., passed entirely via registers), the article shows how to retrieve their values by inspecting the relevant registers or using assembly traces. Example:

void foo(const char* str) {
    char buf[1024] = {'\0'};
    memcpy(buf, str, sizeof(buf));
}
int main(int argc, char* argv[]) {
    foo("abcd");
    return 0;
}

In the compiled binary, the argument str resides in %rdi , and its address can be obtained with x /s $rdi .

Virtual Function Address Errors

The article also covers crashes caused by corrupted virtual function tables. A sample program corrupts the vtable pointer, leading to a core dump at the instruction mov (%rax), %rax . By examining the assembly and the vtable entries, the root cause is identified.

Summary of Core‑Debugging Workflow

Determine the high‑level cause (hardware vs. program).

Locate the offending source line.

Identify the exact instruction that caused the fault.

Inspect the variables involved in that instruction.

Effective use of assembly inspection and GDB printing commands ( x , print , display ) greatly accelerates core analysis.