Simple Testing Can Prevent Most Critical Failures: Findings from an Analysis of Five Open‑Source Distributed Systems

The article discusses the paper *Simple Testing Can Prevent Most Critical Failures*, which analyzes bugs and failures in five widely‑used open‑source distributed systems—Cassandra, HBase, HDFS, MapReduce, and Redis—covering incidents from 2010 onward.

Key conclusions include:

Conclusion 1: 77% of errors require more than two input events (e.g., read/write, node down, network fault), so test cases should combine multiple events.

However, 90% of errors can be reproduced with three or fewer events, easing test design.

Conclusion 2: The order of multiple events affects 88% of errors, necessitating tests that explore different event permutations.

Conclusion 3: Almost all errors (98%) can be reproduced with three or fewer nodes, and 84% with just two nodes, contradicting the belief that large‑scale clusters are required to expose bugs.

Conclusion 4: 74% of errors are deterministic—given a specific event sequence they will always manifest.

Conclusion 5: 53% of nondeterministic errors (19% of the total sample) stem from internal timing issues, such as network glitches.

Conclusion 6: 76% of errors have clear log information, with 84% providing logs that record all triggering events; the average log length is 824 lines, underscoring the need for concise, informative logging.

Conclusion 7: 77% of errors can be reproduced by unit tests, yet many open‑source projects still lack sufficient unit coverage.

The paper also highlights that 92% of crash‑inducing errors arise from poor error‑handling practices, such as single‑line logs without handling, reckless assertions, or placeholder TODOs.

These findings suggest building a centralized error database and applying big‑data analytics to improve testing strategies and error handling in distributed storage systems.

Images illustrating the statistical results are included in the original article: