Using JaCoCo for Test Coverage in Vivo's Internal Development Platform

This article, authored by the Vivo Internet Server Team (Xu Shen), introduces the practice of using JaCoCo to implement test coverage measurement within Vivo's internal development platform. It covers the principles of JaCoCo, the challenges encountered during implementation, and solutions for coverage data loss caused by frequent releases.

Why test coverage is needed

In daily development, test cases are often designed based on experience, leading to insufficient scenario coverage and bugs after release. Unplanned code changes and refactoring can also cause missed test scenarios, making it difficult to quantify testing effectiveness.

Code coverage metrics—function/method coverage, branch coverage, condition coverage, and line coverage—help quantify how much of the source code is exercised by tests.

JaCoCo in test coverage

JaCoCo is the mainstream tool for Java code coverage (C/C++ uses Gcov, JavaScript uses Istanbul). It provides a lightweight, flexible library that can be integrated with various build and development tools.

Advantages of JaCoCo include support for instruction (C0), branch (C1), line, method, class, and cyclomatic complexity analysis; operation on bytecode without source files; low runtime overhead; a comprehensive API; and remote protocol/JMX control for data download.

JaCoCo principle

JaCoCo inserts probes into Java bytecode on‑the‑fly. These probes record execution without altering program behavior. By inserting probes at control‑flow edges, JaCoCo can determine which instructions have been executed.

Example bytecode and probe insertion diagrams illustrate how probes are added to methods and how execution of edges is inferred.

CICD platform solution

The CI/CD pipeline enables test coverage by downloading the JaCoCo agent package and configuring the JavaAgent parameter during Java process startup. The agent intercepts class loading, instruments classes with probes, and records execution data in memory.

During testing, executed code is recorded by probes. After testing, the platform downloads (dumps) coverage data via JaCoCo API, merges historical data, and generates coverage reports. These reports help identify uncovered scenarios for additional testing.

Problems encountered and solutions

1. Inconsistent class IDs across machines : Different compilation environments produce different class hashes, causing mismatched coverage data. The solution is to keep the compilation environment consistent or compile once and reuse the same class files.

2. Coverage of incremental code : JaCoCo does not natively support incremental code coverage. The team proposes calculating branch differences (e.g., via Git diff) and extending JaCoCo to collect separate metrics for new classes, methods, lines, and instructions.

3. Class ID loss when code changes : Modifications can change probe counts and locations, making it impossible to map old probes to new ones. A possible mitigation is to store coverage at the method level rather than the class level, preserving data for unchanged methods.

Various diagrams illustrate these issues and proposed redesigns.

Conclusion

Implementing test coverage with JaCoCo has clearly improved test quality, though it also introduced additional effort for testers. Nevertheless, it encourages deeper code understanding and better collaboration between testers and developers. Test coverage should be used wisely as one of several metrics to enhance overall testing quality.