MDAP Stack Symbolization Service: Architecture, Implementation, and Optimization

Background

MDAP (Multiple Dimension Analysis Platform) is a real‑time multi‑dimensional monitoring and analysis platform that supports custom metrics from business applications. It provides specialized monitoring for mobile app performance data to meet growing data analysis demands.

This article is the fourth in the MDAP series. The previous articles covered the overall design of the observability platform, machine‑learning‑based URL client monitoring, and Android ANR analysis.

Table of Contents

目录

1. 背景
2. 方案调研
3. 堆栈还原服务实现
    3.1 iOS 篇
    3.2 Android Native 篇
    3.3 Android Java 篇
    3.4 Web & React Native 篇
    3.5 痛点与优化
    3.6 Benchmark
4. 未来规划

1. Background

During debugging and error investigation, developers often locate the source of an exception by analyzing the call stack. MDAP’s mobile performance analysis relies heavily on stack collection and symbolization, as illustrated in the diagram below.

The highlighted blue parts require the MDAP SDK to collect stack frames and upload them to the backend, which means the server must support high‑performance symbolization for Android, iOS, Web H5, and React Native stacks.

1.1 Basic Concept of Stack Symbolization

A call stack represents the hierarchy of function calls at a specific moment. Each stack frame (or stack frame ) contains the function name, parameters, file name, and line number. Symbolization (or stack symbolization ) converts raw stack frames into source‑level function calls.

For compiled languages like Java, the runtime may already output source‑like stack traces, but code is often obfuscated during packaging, so a de‑obfuscation step is still required.

1.2 Stack Classification

Four client platforms are considered:

Android

iOS

Web H5

React Native

Android is special because it supports both Java/Kotlin and native C++ code, which have distinct stack formats ( Android Java vs Android Native ).

Stacks can be divided into two categories based on the restoration principle:

Address‑based mapping (Android Native, iOS)

Symbol‑based mapping (Android Java, Web H5, React Native)

Address‑Based Mapping Characteristics

Compiled languages (C/C++)

Produces executable binaries

Stack frames contain return addresses; source information must be derived from addresses.

Symbol‑Based Mapping Characteristics

Interpreted languages (JavaScript, Java bytecode)

Runtime provides function names and line numbers, but they are often obfuscated.

2. Solution Research

MDAP requires two basic capabilities for stack symbolization:

High performance – the platform must handle tens of thousands of QPS during peak traffic.

Standardized, unified architecture – to support multiple stack types with a single model.

Existing tools are usually client‑side utilities that are unsuitable for server‑side high‑throughput processing because of process‑creation overhead, large symbol files, and heterogeneous runtime dependencies.

2.1 Existing Tool Survey

Most tools are designed for local development and cannot be directly wrapped as a backend service due to performance, deployment, and observability concerns.

2.2 Industry Practices and Architecture Design

Two common patterns were identified:

Pre‑parse symbol files and cache the result.

Store the parsed data as KV pairs in Redis or a KV database (e.g., HBase).

This approach separates symbol parsing from stack processing, reducing I/O and improving latency.

2.3 Proposed Architecture

The architecture includes two main services:

SymbolManager : Handles symbol file upload, parsing, and caching.

Stack Symbolicating : Provides gRPC‑based stack symbolization by looking up symbols in Redis.

3. Implementation of Stack Symbolization Service

3.1 iOS

iOS stacks belong to the address‑based category. A typical raw iOS stack frame looks like:

CoreFoundation  0x00007fff20422faa 0x7fff20311000 + 1122218 [8c17697f-2e84-39e5-b491-fcf5169106ff]
libobjc.A.dylib 0x00007fff20193ff5 0x7fff20174000 + 131061 [8c17697f-2e84-39e5-b491-fcf5169106ff]
...

Each line contains the binary image, return address, load address, offset, and UUID.

3.1.1 Symbol Files (dSYM)

iOS uses dSYM files (Mach‑O binaries) that embed DWARF debugging information. DWARF provides compile units, subprograms, and line tables.

3.1.2 DWARF Debug Info Example

0x00195e54: Compile Unit: length = 0x000003ee, format = DWARF32, version = 0x0004, abbr_offset = 0x0000, addr_size = 0x08
0x00195e5f: DW_TAG_compile_unit
    DW_AT_producer    ("Apple clang version 12.0.5 (clang-1205.0.22.11)")
    DW_AT_language    (DW_LANG_ObjC)
    DW_AT_name    ("/Users/.../HamsterDemoCrashViewController.m")
    ...
0x0019604b:   DW_TAG_subprogram
    DW_AT_low_pc    (0x000000010000c8f4)
    DW_AT_high_pc   (0x000000010000cab0)
    DW_AT_name  ("-[HamsterDemoCrashViewController tableView:didSelectRowAtIndexPath:]")
    ...

The restoration process matches a stack address to a subprogram, then retrieves the source file and line number from the DebugLine section.

3.1.3 Address Conversion

MDAP SDK reports both the runtime address and the load address. The file address used for DWARF lookup is calculated as:

file_addr = runtime_addr - load_addr + text_vm_addr

During symbol parsing, this conversion is pre‑computed, so the service only needs to use the relative offset provided in the stack frame.

3.2 Android Native

Android Native stacks are also address‑based and use ELF .so files containing DWARF info. A typical stack line:

pc 0x0000000000022074 libc.so [arm64-v8a::77e6f9ea7bad92cd845bdfb83dcb29d9]

The restoration flow is identical to iOS, with the same SymbolManager and Stack Symbolicating services.

3.3 Android Java

Android Java stacks belong to the symbol‑based category and require de‑obfuscation using the mapping.txt file generated by ProGuard/R8.

Example of an obfuscated stack frame:

tg.a.e(MethodRecorder.kt:1)
 tg.a.f(MethodRecorder.kt:2)
 com.***.apm.trace.method.MethodTracingModule.storeCurrentRecords(MethodTracingModule.kt:1)

The mapping file contains entries such as:

com.***.hamster.similate.CrashActivity -> com.***.hamster.similate.CrashActivity:
    java.util.HashMap _$_findViewCache -> a
    1:1:void
():10 ->
android.view.View _$_findCachedViewById(int) -> a
    1:1:void access$catchFunc(com.***.hamster.similate.CrashActivity):10 -> a
    ...

Restoration steps:

Locate the class block by the obfuscated class name.

Match the obfuscated method name within the block.

Use the line‑number range to select the correct original method.

Calculate the original line number based on the offset.

3.4 Web & React Native

Both platforms use JavaScript source maps. A typical minified stack line:

at n.capture (https://example.com/static/js/207.e017fea1.chunk.js:2:3768321)
at https://example.com/static/js/main.f6b17586.chunk.js:1:17348

Source maps are JSON files containing version , sources , names , and a VLQ‑encoded mappings field. The restoration algorithm:

Find the source map by the generated file name (ChunkHash).

Locate the line entry in mappings using the reported line number.

Decode the VLQ segments to obtain column ranges and source positions.

Match the reported column to a segment and retrieve the original source file, function name, and line/column.

KV Design for Source Maps

Positions are bucketed by start column. The KV key is app_name + js_name + line + bucket_id . The value stores all segments for that bucket as JSON. During symbolization, the service fetches the bucket, performs a binary search on column ranges, and returns the original location.

3.5 Pain Points & Optimizations

Symbol KV caching in Redis faces eviction challenges. To avoid cache misses, a persistent KV store (Shopee’s COPI2, a Redis‑compatible RocksDB service) is introduced, allowing the service to fall back to persistent storage without re‑parsing symbol files.

3.6 Benchmark

Deployed on a 4‑core, 4 GB container (Intel Xeon Silver 4216), the service achieved the throughput shown in the benchmark chart.

4. Future Plans

Key directions include:

Managing system and third‑party library symbol tables to improve overall stack resolution quality.

Aggregating massive stack data to extract common patterns, linking them to code commits, and automatically assigning owners for faster incident response.

Author: Weizhe, MDAP joint project backend engineer, Shopee Engineering Infrastructure.