Design and Implementation of Multi‑Cluster HPA Metrics Collection, Analysis, and Reporting in Kubernetes

Background

The Horizontal‑Pod‑Autoscaler (HPA) in Kubernetes automatically scales workloads based on CPU, memory, or custom metrics, which is especially important in multi‑cluster deployments where cluster weights can be manually or automatically adjusted to reflect real‑time resource availability.

Benefits of HPA

Improves resource utilization.

Reduces manual operational effort.

Provides clear visibility into workload resource demands for better operational decisions.

Measurement Indicators

Key metrics include scaling up/down counts, upper/lower limit adjustments, HPA thresholds, minimum/maximum replica counts, and peak/low‑peak CPU usage statistics.

Data Collection

A table tb_hpa_xxxx is created to store HPA metrics (uc, dc, maxuc, mindc, timestamps, etc.).

create table tb_hpa_xxxx(
  id SERIAL PRIMARY KEY,
  appcode varchar(256),
  uc int DEFAULT 0,
  dc int DEFAULT 0,
  maxuc int DEFAULT 0,
  mindc int DEFAULT 0,
  create_time timestamptz NOT NULL DEFAULT now(),
  update_time timestamptz NOT NULL DEFAULT now()
);
COMMENT ON TABLE tb_hpa_xxxx IS 'HPA指标收集';
...

SQL queries are defined to aggregate scaling events, high‑peak and low‑peak CPU usage, and pod counts per application code.

select G.*, N.min_replicas, N.max_replicas
from (
  select A.deployment_base as env, A.appcode, A.annotations as hpa,
         coalesce(M.uc,0) as uc, coalesce(M.dc,0) as dc,
         coalesce(M.maxuc,0) as maxuc, coalesce(M.mindc,0) as mindc
  from (... ) A
  left join (... ) M on M.appcode = A.appcode and M.env_name = A.deployment_base
) G
left join tb_k8s_appcode_hpa N on G.appcode = N.appcode and G.env = N.deployment_base;

Python Implementation

The HpaReport class implements methods to execute the above SQL statements, compute high‑peak and low‑peak statistics, and store the results in a reporting table tb_hpa_report_xxx . It also generates an HTML email with a detailed table of scaling counts, CPU usage percentiles, and pod numbers.

class HpaReport(Base):
    @try_catch_db_exception
    @commit_on_success
    def stats_hpa_updown(self, start_time, end_time):
        rows = db.session.execute(text("""
            select G.*, N.min_replicas, N.max_replicas ...
        """), {"start_time": start_time, "end_time": end_time})
        return self.rows_as_dicts(rows.cursor)

    def save_stats_result(self, day):
        hpa_stats_rows = self.stats_hpa_updown(...)
        hcpu_stats_rows = self.stats_high_time_cpu(...)
        lpods_stats_rows = self.stats_low_time_pods(...)
        # merge results and insert into tb_hpa_report_xxx
        for value in report_rows.values():
            model = HpaReportModel(record_time=day, **value)
            db.session.add(model)

Report Generation and Validation

The send_report_form method builds an HTML table summarizing daily HPA scaling events, CPU usage percentiles, and pod counts, then emails it to stakeholders. Validation steps compare the number of reported records with the actual HPA‑enabled applications and verify that scaling counts and CPU statistics match raw metrics.

Conclusion

Accurate data collection, cleaning, and aggregation are essential for reliable HPA reporting; without proper configuration, HPA may not reduce resource usage, especially when minimum replica settings are too high.