Scalable System Design Best Practices – Lessons from Dropbox Operations

Dropbox operations engineer Rajiv presents the first lecture on scalable system design best practices. Dropbox serves 40,000,000 users while its operations team consists of only one to three engineers.

Run with extra load (Discover system failures through additional load)

A common production technique is to artificially generate extra data load, such as additional Memcached reads, to quickly detect failures. Simulating write load is discouraged because it can corrupt data consistency and cause uncontrolled lock contention.

App‑specific metrics

Aggregating custom metrics across clusters is essential. Dropbox combines Memcached, cron jobs, and Ganglia: metric data is stored in a thread‑safe memory block, sent to Memcached every second with timestamps as keys, and aggregated each minute for monitoring. An example chart shows response time breakdown by component.

Figure 1: System response time metric chart

The X‑axis is time, the Y‑axis is server response time divided into MySQL Query, MySQL Commit, RPC, Memcached, and CPU. A spike around 1:00 is caused by MySQL Commit.

Poor man’s analytics with Bash

Proficient use of Bash can greatly improve efficiency. For ad‑hoc analysis of recent traffic peaks, the following script extracts timestamps from logs, counts occurrences, and plots them with gnuplot:

Apr 8 2012 14:33:59 POST ... Apr 8 2012 14:34:00 GET ... Apr 8 2012 14:34:00 GET ... Apr 8 2012 14:34:01 POST ... cut -d' ' -f1-4 log.txt | xargs -L1 -I_ date +%s -d_ | uniq -c | (echo "plot '-' using 2:1 with lines"; cat) | gnuplot

This command visualizes the current system state.

Log spam is really helpful

What appears as noisy logs can be valuable for tracing code paths; maintaining both clean and noisy log files helps locate issues when they arise.

Keeping a downtime log

Recording start/end times and causes of incidents enables objective analysis to minimize future downtime.

UTC (Use Coordinated Universal Time)

Always store server and database timestamps in UTC to avoid timezone‑related bugs; convert to local time only when presenting data to users.

Technologies we used

Dropbox’s production stack includes:

1) Python

2) MySQL

3) Paster/Pylons/Cheetah web framework

4) Amazon S3/EC2

5) Memcached

6) Ganglia

7) Nginx

8) HAProxy

9) Nagios

10) Pingdom

11) GeoIP

The choices favor reliability and low risk; even widely used tools like Memcached have quirks, so newer, untested technologies are avoided.

The security‑convenience tradeoff

Increasing security often reduces user convenience, such as generic error messages that hide which credential is wrong. Internal firewalls are useful, but may be omitted for isolated server clusters. Security decisions should be weighed against actual necessity.