Evolution of CI at Laiye Technology: From Manual Processes to Cloud‑Native Prow and Tekton

Laiye Technology started with a modest micro‑service landscape of about 30 services, relying on manual updates for development and a mix of manual steps and Python scripts for production deployments. This approach proved inefficient as service count grew.

To improve efficiency, the team introduced a semi‑manual stage using GitLab for source control and Jenkins for CI, creating one Jenkins freestyle job per project. While production deployments became automated, development still required manual code pulls and builds, limiting scalability.

In the early automation stage, each service received its own Jenkins job, consolidating build and deployment but increasing configuration overhead. The team then adopted Declarative Pipelines and shared pipeline libraries, reducing per‑service configuration and separating build from release, with production deployments triggered manually after tag creation.

Facing the challenges of a cloud‑native environment with over 250 services, Laiye migrated code repositories to a privately hosted GitHub and adopted Prow as the primary CI engine, extending it with external plugins and Tekton for Kubernetes‑native pipeline execution.

Prow provides GitHub automation, ChatOps commands, OWNERS‑based permission control, automatic PR merging, label management, branch protection, release notes, parallel CI execution, Prometheus metrics, and Config‑as‑Code. The company customized Prow for code review, automatic merging, and post‑merge testing, integrating presubmit and postsubmit jobs, periodics for scheduled tasks, and external plugins for image updates and security checks.

The CI workflow includes static analysis, image building with multi‑stage Docker builds, automated Helm deployments to development and test environments, integration testing via Prow and an internal platform (Siber), security scanning before production release, and conditional production deployment based on test results.

Branch strategies follow Gitflow: feature branches for development, test branches for QA, master for release with semantic version tags, and hotfix branches for urgent patches.

Configuration management remains file‑based rather than using a central config store. The team introduced templated configuration files with environment‑specific fill‑ins, generating ConfigMaps that are applied via GitOps (ArgoCD) for non‑production environments and manually for production, with automated validation and review steps integrated into the CI pipeline.

Overall, the transition from manual processes to a comprehensive, cloud‑native CI system has dramatically increased efficiency and reliability, even with a single SRE engineer maintaining the pipeline.