Building the Most Reliable Autonomous Driving Infrastructure at Pony.ai

The talk focuses on building the most reliable autonomous driving infrastructure at Pony.ai, covering system architecture, technical challenges, and practical solutions.

Traditional internet company infrastructure includes large‑scale databases, distributed file systems, compute platforms with many servers, big‑data platforms, container management, and continuously evolving web services.

For autonomous driving, Pony.ai adds several layers: an onboard vehicle system that supports diverse AI algorithms and vehicle control, a massive simulation platform that runs at least 300,000 km of virtual testing per day, a fleet operation platform for robotaxi services, and a visualization plus human‑machine interface for monitoring and passenger interaction.

Rapid growth introduces challenges such as increasing vehicle numbers, expanding operational areas, petabyte‑scale data growth, and a growing engineering workforce, all demanding a highly scalable technology stack.

The autonomous driving stack consists of sensors and hardware (LiDAR, cameras, radar, GNSS/IMU, compute), sensor fusion, perception, behavior prediction, decision‑making, control, high‑definition maps and localization, and data flow management.

Pony.ai developed its own vehicle‑mounted system, PonyBrain, which handles multi‑module scheduling, inter‑module messaging, resource allocation (CPU, GPU, memory), comprehensive logging, and monitoring with alerting to ensure safety.

Reliability challenges: zero tolerance for memory leaks or logic errors, strict code reviews, extensive unit testing, static analysis, ASAN, multi‑stage checks (pre‑launch validation, runtime monitoring, post‑run analysis).

Performance challenges: low‑latency communication, avoiding costly data copies, appropriate resource assignment, regular profiling.

Flexibility challenges: generic module interfaces and messaging to support varied compute resources and rapid iteration.

The simulation platform provides reliable results by modeling vehicle dynamics on the server side, selects meaningful road‑test data automatically, supplements with synthetic scenarios, and runs on a distributed computing platform to achieve the required throughput.

Data infrastructure must store petabyte‑scale data, enable fast access, support processing pipelines, synchronize data across regions, handle cold and hot data differently, ensure high availability, allow horizontal scaling, and control costs.

Data processing challenges include minimizing end‑to‑end latency, selecting CPU‑intensive or I/O‑intensive pipelines based on task type, and defining extensible tasks for new data analyses.

The fleet operation platform comprises a Fleet Control Center, the Pony Pilot app, the onboard system, and various web applications for monitoring and managing vehicles and personnel.

Fleet platform challenges involve supporting a complex, evolving web framework, managing numerous web services, providing deployment tools, logging platforms, and monitoring for high availability.

Container and service scheduling is handled via Kubernetes.

The visualization platform aims to make the autonomous vehicle’s perception understandable to humans, requiring flexibility, high‑performance 3D rendering, and cross‑platform support (desktop, mobile, web).

The human‑machine interface offers passengers an informative UI that displays the vehicle’s perception, decisions, and planned actions, enhancing trust.

In summary, Pony.ai’s infrastructure combines traditional internet challenges with autonomous‑driving‑specific requirements, offering engineers the chance to work on all aspects of self‑driving systems, design and implement scalable distributed solutions, and drive continuous technological advancement.