How to Verify the Correctness of Probabilities in Game QA

As QA professionals, we must use factual evidence to verify probability settings in games, turning vague player rumors about "lucky draws" into measurable data.

After understanding how random numbers are generated, the key question is how to test a probability defined by a designer. Using a black‑box approach, we automate attacks in a PC game, record each hit type (critical, normal, dodge), and collect 50,000 samples to compare actual rates with expected ones.

Statistical results show that the observed critical and dodge rates closely match the expected probabilities, and larger sample sizes reduce error, but raw frequencies alone do not guarantee correct randomness.

For example, if all 885 critical hits occur in the first 885 attacks and none thereafter, the overall rate still matches expectations, yet the distribution is clearly non‑random.

To assess distribution, we examine the intervals between successive events. By counting how often a given distance (e.g., 0, 2) occurs, we can plot a histogram that reveals clustering or long gaps, providing insight into whether the random generator behaves properly.

When dealing with more complex random rules—such as generating five attributes that must satisfy min/max bounds and sum to a target value—designers may only specify a vague goal like “approximately normal distribution.” In such cases, we collaborate with programmers and designers to iteratively adjust the algorithm, collecting statistical data (e.g., 10,000 samples) and visualizing attribute value frequencies.

The charts expose anomalies, such as an unexpected spike at the maximum value (30) and a desire to center the distribution around 12. After further tuning, the final distribution aligns better with the intended normal shape.

Finally, we also examine the frequency of a specific attribute value (e.g., 12) across 11,000 samples to ensure the distribution meets expectations.

In summary, whether dealing with simple or complex random mechanics, QA should combine raw probability checks with distribution analysis, using statistical visualizations to provide concrete evidence for developers and designers, and to decide if the implemented randomness meets the design intent.