Small Data vs. Big Data: How Minor Signals Guide Robust Data Management

Editor’s note: Both Ma and Dr. Wang Jian have said that data is the future’s “new energy” that will drive society forward. Professor Bao Yongjian, an associate professor at the University of Lichao’s School of Management and a distinguished EMBA professor at Fudan University, also likens data management to mining.

“Small data are filtered and ignored because people lack concepts to define and explain them. Yet without small data, big‑data management falls into traps. Prioritizing small data and treating big data as a servant is the proper path of data management.”

Big‑data management mines massive datasets to uncover hidden variables and causal links, enabling targeted production and marketing. In contrast, small data refer to scattered weak signals that often appear as random noise or unexplained deviations.

Small data are filtered and ignored because people lack ready concepts to define them; without them, big‑data management is full of pitfalls.

Small data as the master, big data as the servant—that is the correct way of data management.

Big‑Data Pitfalls

Consider a driver who has driven thousands of trips accident‑free, drinks a little at a friend’s house, and decides to drive home, assuming the probability of an accident is only one in a thousand. This is a sampling error: the previous thousand trips without alcohol cannot be mixed with the current risky trip. This mirrors a common mistake in big‑data “mining”.

From the first Super Bowl in 1967 to the 31st in 1997, whenever an NFL team won, the stock market rose over 14%; when an AFL team won, it fell at least 10%. Betting on such indicators would be hazardous. In 1998 Denver Broncos (AFL) won and the market surged 28%; in 2008 the New York Giants (NFL) won, the market plunged 35% and triggered the sub‑prime crisis.

With enormous samples and many variables, we can find absurd correlations that satisfy statistical rigor but lack causal meaning. The U.S. government publishes 45,000 economic indicators annually; one could generate billions of hypotheses about what influences unemployment or interest rates. Repeatedly testing models eventually yields statistically significant correlations, yet treating them as causation is another big‑data mining trap.

We say a three‑foot‑deep pool can drown a person because three feet is only an average depth. Ignoring extremes and relying on averages is a third common pitfall.

John von Neumann once joked, “Give me four parameters and I can draw an elephant; add a fifth and I can make its trunk stand up!” Big‑data mining may produce novel correlations, but without contextual grounding they can mislead decisions.

Small Data Holds the Golden Key

Roger Barnsley noticed that children born under Capricorn, Aquarius, and Pisces seemed naturally inclined to ice hockey, prompting a study of Canadian adult team players’ birthdays that revealed the same pattern. The explanation lay in the sport’s age‑group cutoff of January 1, giving those born in January‑March a developmental advantage.

Malcolm Gladwell’s book “The Tipping Point” demonstrates how small data can decode major phenomena.

Minor details unlock big problems. For example, San Francisco health officials predicted rising AIDS cases based on increasing hepatitis rates among gay men, but the forecast failed because the community’s behavior changed—people began posting health status online to avoid cross‑infection, a nuance only small‑data analysis captured.

Small data are hard to find because they are weak signals that occur infrequently and are often buried in outliers. People tend to dismiss unfamiliar weak signals as background noise, which can lead to serious misjudgments, such as overlooking a terrorist’s plan to learn only how to fly an aircraft.

Douglas Hubbard suggests three ways to work with small data:

1) Do not seek perfection; aim for continual improvement. The ancient Greek Eratosthenes estimated Earth’s circumference using sun angles at two locations—his result had large error but represented a leap in knowledge.

2) Use “stepping‑stone” knowledge through analogies. Physicist Enrico Fermi asked how many piano tuners were in Chicago, guiding students to estimate populations, households, and piano ownership to arrive at a plausible figure.

3) Avoid over‑simplifying premises, yet strive for simple models. Nine‑year‑old Emily Rosa designed a straightforward experiment to test “energy fields” of healers, publishing her findings in JAMA and becoming the journal’s youngest author.

These three steps—incremental refinement, analogical borrowing, and simple modeling—represent the best use of Bayes’ Theorem, the core of small‑data reasoning.

Master and Servant

Complex systems, such as weather forecasting, exhibit three typical traits: many variables, interactions, and simultaneous occurrence. In 1916, Lewis Fry Richardson attempted to “big‑data‑ify” weather by dividing Germany’s climate into a grid of small matrices, hoping to simplify predictions, but lacked computational power.

By 1950, Norman combined Richardson’s approach with computers, leading to increasingly reliable forecasts.

Richardson’s contribution was extracting key small‑data insights from climate phenomena; Norman’s contribution was using big‑data computation to model those insights dynamically. Their synergy enables modern weather prediction.

Integrating the strengths of both big and small data allows us to refine cognitive models and distinguish signal from noise, echoing George Box’s famous remark: “All models are wrong, but some are useful.”

(Author: Associate Professor at the University of Lichao School of Management, Distinguished EMBA Professor at Fudan University)