Performance Optimization of Qunar's ATPCO International Ticket Pricing System

2014 joined Qunar and works in the International Ticket team of the flight ticket division, responsible for the development and maintenance of the main site, fare publishing, and surrounding systems, with a focus on performance optimization and JVM tuning.

ATPCO is Qunar's fare system that receives pricing and rule data from the US ATPCO company for nearly 500 airlines, providing fare search services to the upper‑level ticket search system. The base data includes about 90 million fare records and 120 million rule records, with each fare linked to roughly 20 rules, making the business rules complex and the data volume large.

For a single‑way search from Beijing to Hong Kong on 2018‑07‑01, the estimated computation involves 5 routing points, 500 fares, 20 flights, 10 cabins, and 2 cabin rules, resulting in a total of 1,000,000 calculation units per request, which caused high CPU queue lengths even on a 24‑core machine.

Consequently, the system initially responded slowly, with an average single‑way latency of 500 ms and round‑trip latency up to 1500 ms, sometimes timing out.

Problem analysis identified four main causes: (1) code bottlenecks such as lock contention and inefficient data fetching; (2) excessive computation volume leading to thread timeouts; (3) unreasonable processing order causing CPU imbalance; and (4) other factors like GC pauses and slow external dependencies.

To address these, the team applied two classic principles: Little's Law for reducing work‑in‑process and Amdahl's Law for improving parallelism.

Little's Law practice – Little’s Law (formulated by MIT professor John Little in 1961) relates throughput, work‑in‑process, and cycle time, guiding the two optimization directions of increasing per‑unit efficiency and decreasing task count.

Based on Little’s Law, the optimization focused on (1) improving code performance and (2) pruning the computation.

Code performance improvements included:

Reducing string usage by converting many strings to int/long, lowering memory consumption and increasing operation speed.

Avoiding low‑performance APIs (e.g., replacing String.split with Guava Splitter, preferring BigDecimal.valueOf over new BigDecimal , and pre‑allocating capacities for collections and StringBuilder).

Minimizing I/O blocking and lock waiting by cutting unnecessary log output and configuring Logback with neverBlock=true to prevent log‑induced thread stalls.

Improving cache efficiency by switching from Guava Cache to Caffeine, which offers several‑fold higher read/write performance in read‑heavy scenarios.

Pruning (剪枝) – After exhausting code‑level optimizations, the team reduced the combinatorial explosion of round‑trip flight pairs by sorting flights by price and selecting a subset of return flights, cutting the number of flight combinations by over 50% and reducing timeout occurrences by more than 30%.

Amdahl's Law practice – Amdahl’s Law states that the overall speedup of a system is limited by its serial portion. The team therefore:

Improved concurrency efficiency by replacing a regular thread pool with a ForkJoinPool , leveraging work‑stealing to balance load across CPU cores.

Enabled parallel processing by distributing independent computation types (Specified fare, Constructed fare, End‑on‑end) across multiple machines, increasing overall throughput while managing the added scheduling complexity.

Switching to ForkJoinPool balanced CPU usage and further reduced round‑trip timeout rates, as shown by post‑optimization mpstat graphs.

Overall, the combination of Little’s Law‑guided pruning, code refactoring, cache replacement, and Amdahl’s Law‑based concurrency enhancements significantly improved the ATPCO system’s response time, CPU utilization, and reliability.

Related links: 1. Caffeine GitHub: https://github.com/ben-manes/caffeine 2. Caffeine principles: https://segmentfault.com/a/1190000008751999