Measuring Engineering Effectiveness: A Four‑Type Theory of Software Quality

We discuss engineering effectiveness measurement, dividing it into two parts: engineering efficiency (speed and quality) driven by the development team, and effectiveness (usability) driven by product managers, UX designers, and engineers.

Google engineers have published an IEEE article with similar viewpoints, emphasizing that developer productivity consists of speed, usability, and quality, and that measuring all three is necessary to avoid unintended trade‑offs.

A literature review identifies seven recurring dimensions of code quality: defect rate, reliability, maintainability, testability, complexity, understandability, and readability.

Two rounds of interviews with Google engineers reveal that developers define code quality primarily in terms of maintainability, with testability, understandability, complexity, and readability as sub‑aspects, while product quality is seen as usability, practicality, reliability, and (interestingly) not innovation.

Based on these findings we propose a “quality theory” consisting of four types of quality—process quality, code quality, system quality, and product quality—each influencing the next (process → code → system → product).

Process quality is signaled by thorough testing, code reviews, consistent planning, and predicts overall software quality better than traditional code metrics.

Code quality, viewed by developers as ease of understanding and modification, impacts reliability and defect rates but does not guarantee them.

System quality aggregates reliability, performance, and low defect rates, yet is hard to measure due to sparse data (e.g., rare service interruptions).

Product quality is ultimately judged by customers and includes practicality, usability, and reliability; high system quality is a necessary but not sufficient condition for high product quality.

We highlight measurement challenges, such as data sparsity and Goodhart’s law, and suggest that technical leaders adopt the four‑type model to align stakeholders around a common definition of software quality and to select appropriate intermediate metrics.