Google’s Project Health: Evolving Test Certified into a Comprehensive Engineering Productivity Framework

This article was posted by Guowen (working at Google) on the Testerhome community’s public account.

Google has turned the Test Certified program, launched in 2007, into a more comprehensive “Project Health” system that leverages changes in engineering measurement data to discover ways to boost engineer productivity.

The transformation was possible because Google spent more than a decade building a suite of production tools that both provide a good developer experience and collect abundant real‑world data, forming a complete toolchain.

These toolchains embody Google’s unique “engineering consistency”.

For domestic companies, the principle of “improving engineering consistency” should be combined with tooling support, gradually improving processes; see DevOps in a Hundred‑Person Organization: The First Three Steps .

Recent years have seen a surge of efficiency‑focused R&D initiatives across major companies, with MTSC conferences regularly sold out. A few years ago I wrote about Google’s Test Certified; now I discuss Project Health (PH).

Overview

The overall goal is to improve engineering productivity by discovering problems and inefficiencies early in the software lifecycle, covering development, testing, release, and deployment.

Project Health Metrics

Project health metrics are used to calculate a project’s health level and are usually divided into the following categories:

Direct metrics: whether tests exist and are in use.

Efficiency: are tools and processes effective? (Engineers are unlikely to use slow tools frequently.)

Negative impact: are there problems that could be prevented by early detection?

PH Level

Internal tools define an automatically collectible metric set that can display the overall project level as well as the level of each specific metric.

Presubmit Tests

Testing can catch problems at the earliest stage—before code is submitted. Teams should encourage running presubmit tests and using small, fast, isolated unit tests, because long wait times cause developers to skip them.

tap-greenness – CI pass rate. A high green rate indicates well‑maintained unit tests and reduces developer frustration.

tap-flakiness – CI instability rate. Frequent flaky tests waste resources and erode trust, making real failures harder to spot.

presubmit-coverage – Presubmit test coverage, expected to be high because most presubmit tests are unit tests.

presubmit-ignored – Rate of ignored presubmit tests; lower is better.

presubmit-latency – Time taken to run presubmit tests before a commit.

Test Coverage

absolute-coverage – Overall test coverage.

incremental-coverage – Coverage of newly added code.

Releases

release-duration – Average time a release cycle takes, including manual steps.

release-granularity – Number of code changes per release; smaller releases mean fewer changes per cycle.

release-cherrypick-count – Number of patch‑back operations; ideally releases go smoothly, but this metric tracks extra patches or rollbacks.

GCP Project Overview

Test Certified vs. Project Health

Project Health is a subset of the Test Certified standards, focusing on automatically calculable metrics and data sources to assess a product or project’s health. To obtain a more complete status, additional information sources must be gradually introduced.

References

Google Cloud Platform practice: GCP – Move Fast and Don’t Break Things (Cloud Next ’19) PPT: https://speakerdeck.com/ankitmehta/gcp-move-fast-and-dont-break-things Video: https://www.bilibili.com/video/BV1hf4y1y7Gu/

Related Reading:

DevOps Speaker Joe Liang: Key Insights