Using the crash Tool to Analyze Android Kernel Kdump Dumps

Kdump is a Linux kernel crash‑capture mechanism that saves the system state when a fatal error occurs, enabling post‑mortem analysis. While many tools (e.g., trace32, crash) can parse Kdump dumps, this article focuses on practical usage of the crash utility in the mobile domain, especially on Android devices based on MTK and Qualcomm platforms.

Why use crash? In mobile scenarios trace32 often consumes excessive host memory because it loads the entire dump into RAM. For a 16 GB dump on a PC with 32 GB RAM, trace32 may occupy ~17 GB, leaving little room for other processes. The crash tool employs on‑demand loading, reading only the required dump portions, allowing several terminals to run simultaneously on a modest 6 GB VM.

Dump loading commands

MTK platform: (command shown as an image in the original article)

Qualcomm platform: (command shown as an image in the original article)

For Qualcomm, the command is split into three logical parts:

Specify the physical address of each dump file (extracted from dump_info.txt using grep DDRCS dump_info.txt -nr ).

Provide the KASLR offset ( kimage_voffset ) extracted from OCIMEM.BIN – the little‑endian value 0x0000002821a00000 ).

Calculate vabits_actual (the number of valid virtual‑address bits). For the example it is 39, obtained via the kernel’s extract‑ikconfig tool.

After calculating these parameters, a ready‑made crash.sh script can load the dump automatically.

Basic information collection

When first loading a dump, newcomers often feel lost. The article recommends gathering the following baseline data:

System information (CPU count, uptime, kernel version, memory size, panic reason).

Memory usage.

Interrupt status per CPU.

Run‑queue thread times.

dmesg log.

Trace information (requires the trace.so plugin).

Each item is accompanied by the exact crash command (displayed as images in the source).

Viewing a task’s call stack

For blocked or runnable tasks, the bt command (with space‑separated PIDs) shows the stack. Running tasks, however, often cannot be displayed directly because the stack frame is missing. Three recovery methods are described:

Read the stack pointer (SP) from cpu_context and invoke bt -S <sp+8> .

Extract the SP value from a vendor‑provided .cmm script (e.g., debug.cmm for MTK or corevcpuN_regs.cmm for Qualcomm).

Manually reconstruct the stack frames by locating the deepest valid frame using the frame pointer (FP) chain.

The article also explains ARM64 calling conventions (ATPCS), stack‑frame layout, and how overlapping or differing frame sizes affect back‑tracing, illustrated with several diagrams.

Recovering local variables

Global variables are trivial ( p <symbol> ). Per‑CPU globals require the syntax runqueues@<cpu> . Recovering locals is more involved:

For parameters passed in registers (e.g., x0 ), locate the caller’s register value via disassembly and the saved stack slot.

For true locals stored in registers (e.g., w26 , w27 ), determine whether the register is saved in the current frame; if not, read its value from the vendor .cmm script.

Concrete examples show how to compute the address of x21 (holding msdata ) and retrieve its value, as well as how to obtain newstate and curstate (both equal to 2 in the crash example).

Finding threads that access a specific variable

The search -t <address> (all tasks) or search -T <address> (running tasks) commands locate threads referencing a given memory location. The article demonstrates searching for the address of msdata and identifying an additional thread (pid = 943) that accesses it.

Conclusion

The crash tool complements trace32 by offering low‑memory on‑demand dump analysis and powerful commands (search, foreach, list). It enables recovery of both task and interrupt stacks, inspection of global and local variables, and identification of threads interacting with specific data, making it an essential utility for kernel developers and operators.