Dynamic Runtime Configuration Management at Facebook: Use Cases and Tooling

This article, sourced from Facebook Research, examines the challenges of managing dynamic runtime configuration for large‑scale internet services, where configuration items may be updated multiple times per day without redeploying or restarting applications.

New product feature gating : Facebook releases code early and frequently, using feature switches to keep new functionality disabled until it is ready. The GateKeeper tool incrementally rolls out features and can instantly disable problematic code, targeting specific user groups or device percentages.

Conducting experiments : A/B testing guides data‑driven decisions, such as adjusting VoIP echo‑cancellation parameters per device. Configuration changes enable real‑time experiments in production.

Application‑level traffic control : Configuration drives site‑traffic management, automated traffic shifts across regions, load testing, emergency traffic draining, shadow testing, and fault‑injection drills to assess recovery capabilities.

Topology setup and load balancing : Facebook stores user data in the large distributed store TAO. Changes in hardware, communication patterns, or failures trigger configuration updates that adjust TAO topology and rebalance load.

Monitoring, alerts, and remediation : The monitoring stack is controlled via configuration, specifying which metrics to collect, dashboard layouts, alert detection rules, subscription rules, and automated remediation actions such as server restarts.

All these adjustments can be made dynamically without code changes, enabling rapid troubleshooting and data collection.

Updating machine learning models : Models used for search ranking, news feed ranking, and spam detection are retrained on fresh data and distributed to servers via configuration updates, with payload sizes ranging from kilobytes to gigabytes.

Controlling an application’s internal behavior : Common use cases include configuring storage parameters, cache memory reservations, batch write sizes, and prefetch amounts, all governed by runtime configuration.

The figure shows Facebook’s configuration management ecosystem, which supports the scenarios described above.

Configerator provides core capabilities such as version control, authoring, code review, automated canary testing, and configuration distribution; other tools are built on top of it.

Gatekeeper manages rolling releases of new product features and supports A/B testing for optimal parameter selection.

Package Vessel uses peer‑to‑peer file transfer to distribute large configurations (e.g., multi‑gigabyte machine‑learning models) while preserving consistency guarantees.

Sitevars is an early shim layer offering a simple configuration API for front‑end PHP services.

MobileConfig manages configurations for Android and iOS apps, linking to back‑end systems like Configerator and Gatekeeper, but not to Sitevars or Package Vessel because mobile apps currently do not require massive configuration transfers.

Future articles will dive deeper into Configerator’s powerful features.