How PerRequest Diagnosis Transforms Mobile CDN Issue Resolution

Applicable Scenario

In the mobile Internet context, the article introduces a PerRequest‑level method for precise problem location and diagnosis, aiming to improve the quality of Weibo image access.

Users frequently encounter "white image" issues while browsing Weibo, as shown in the complaint screenshots.

The full request chain includes:

From the mobile device to the network access point (4G/3G/2G or Wi‑Fi gateway).

From the access point to Sina CDN edge nodes (L1 Cache), which are geographically close to the user.

If the image is not cached at the edge, the request is forwarded to the origin server via L2 Cache.

Any anomaly in these links can cause delivery failures. Analysis of error logs for "white images" revealed two clear characteristics:

Feature 1: Error codes concentrate on -1001, -1005 (iOS) and 8009, 8000 (Android).

Feature 2: The mobile client receives the first HTTP response packet but the subsequent packets are incomplete.

Traditional Solutions

Manually repeat browsing on the client to reproduce the issue, which is unreliable due to the sporadic nature of "white images".

Analyze server‑side and client‑side logs extensively. Logs from L4/L7, L1/L2 caches, and the origin are isolated, making it akin to finding a needle in a haystack.

Request user assistance, which often adds noise rather than clarity.

Our Thinking

The shift from PC‑centric fixed networks to mobile Internet introduces three challenges:

Users are constantly on the move, causing network conditions to change at any moment.

Mobile apps rely on heavily abstracted iOS/Android libraries, which obscure low‑level error details.

Mobile operating systems do not provide built‑in packet capture or diagnostic tools.

Proposed Solution

1. Enumerate Error Codes via Controlled Packet Loss

A laptop acts as a gateway, sharing Wi‑Fi with the phone while connecting to the Internet. A custom Linux kernel module precisely drops specific packets (e.g., SYN, SYN‑ACK, or the nth HTTP response packet) to trigger each error code, allowing clear mapping of codes like -1001 and -1005.

2. Build On‑Site Reconstruction Mechanism

Each CDN node adds a custom HTTP response header

Supply-Chain

containing:

The TCP four‑tuple (source IP, source port, destination IP, destination port) for both inbound and outbound connections.

Timing metrics for each processing stage (handshake, request/response intervals, etc.).

The mobile client, upon encountering a "white image", reports the received

Supply-Chain

header together with its error log, forming a closed‑loop view of the entire request path.

Impact of PerRequest Diagnosis

Reduces reliance on user participation for problem reproduction.

Provides accurate on‑site reconstruction in complex mobile network environments.

Minimizes the volume of logs that need to be examined, focusing analysis on targeted client reports.

Real‑world deployment in Sina CDN showed that the four error‑rate metrics became fully explainable, and two of them were reduced by 50% and 30% respectively after targeted optimizations.

Key Takeaways

Adopt a supply‑chain perspective for application delivery quality.

Use precise, per‑request diagnostics rather than exhaustive log mining.

Design architectures with built‑in diagnostic hooks to simplify future troubleshooting.