Mastering GitOps: Build a Full CI/CD Pipeline with GitLab, Argo CD, and Kustomize

Project Overview

This project builds a complete CI/CD pipeline based on the GitOps philosophy to achieve high automation and consistency between development and operations. It integrates GitLab, GitLab Runner (deployed on Kubernetes), Maven, Java, SonarQube, Harbor, and Argo CD to automatically compile, test, scan, build images, update manifests, and perform rolling, blue‑green, canary, and multi‑cluster deployments after each code push.

CI/CD Pipeline Process

Code commit: developers push Java code to the GitLab repository, triggering the pipeline.

Compile and build: GitLab Runner pulls the latest code and uses Maven and the Java toolchain to produce a deployable artifact (Docker image).

Unit testing and code scanning: run unit tests and static analysis with SonarQube to ensure quality and security.

Artifact upload: push the built Docker image to a private Harbor registry.

GitOps deployment: Argo CD watches the Git repository for infrastructure and application changes and automatically applies them to the Kubernetes cluster, using Git as the single source of truth and supporting rolling, blue‑green, canary, and multi‑cluster deployments.

Continuous monitoring and feedback: expose metrics via exporters from GitLab Runner, Argo CD, etc., allowing teams to monitor pipeline status and deployment results in real time.

GitLab CD Disadvantages

Agent permissions are too broad: GitLab Runner often requires cluster‑admin rights, which can become a security risk.

Limited deployment capabilities: GitLab Runner relies on

kubectl

and lacks the advanced features of dedicated tools like Argo CD (auto‑sync, health checks, rollbacks, multiple deployment strategies).

Audit and compliance: Argo CD provides richer audit logs and enforces Git‑only changes, automatically reverting any out‑of‑sync modifications.

GitOps Advantages

Enhanced security: deployments use only Git updates, eliminating the need for Kubernetes or cloud credentials and leveraging Git’s cryptographic guarantees.

Single source of truth: all application and infrastructure configurations live in Git, benefiting from version control, history, audit, and rollback.

Improved development efficiency: familiar Git workflows accelerate iteration, shorten time‑to‑market, and increase system stability.

Simplified compliance auditing: infrastructure changes follow the same pull‑request and code‑review process as software, ensuring transparency.

More GitOps information can be found at the referenced blog posts.

Kustomize vs. Helm

Kustomize emphasizes declarative management and treats configuration as code, allowing layered overlays without templates. Helm is a package manager that uses templated Charts. Key differences include template support (Helm ✓, Kustomize ✗), overlay support (Helm ✗, Kustomize ✓), packaging (Helm ✓, Kustomize ✗), and native Kubernetes integration (Helm ✗, Kustomize ✓).

Although Kustomize lacks packaging and rollback, Argo CD can handle those functions, aligning well with GitOps version‑control principles.

Project Flow Diagram

Preparation Work

Service Deployment

Deploy GitLab, SonarQube, Harbor, buildkitd, and GitLab Runner services. After deployment, optimize the Runner as needed.

Deploy Argo CD and optionally Argo CD Rollouts for blue‑green or canary releases.

Runner Image Build

Build a custom

gitlab-runner-agent

image used in the CI pipeline. Dockerfile:

<code>FROM alpine:latest
USER root
RUN apk update && \
    apk add --no-cache git && \
    rm -rf /var/cache/apk/*
COPY kustomize /usr/bin/kustomize
COPY nerdctl /usr/bin/nerdctl
COPY buildctl /usr/bin/buildctl
# docker build -t harbor.local.com/cicd/gitlab-agent:v1.1 .</code>

Pipeline Image Build

Build Maven, Sonar‑scanner, and JMeter images as needed.

Project Code Repositories

Gitee: https://gitee.com/cuiliang0302/spring_boot_demo

GitHub: https://github.com/cuiliang0302/spring-boot-demo

GitLab Project Permission Configuration

Refer to the linked documentation for detailed permission settings.

Email Configuration

Configure email integration following the referenced guide.

Create CI User and Add to devops Group

Create a

gitlabci

user for committing Kustomize‑updated manifests and assign the maintainer role.

Argo CD Project and Repo Creation

Create a project and repository in Argo CD as shown in the screenshots.

GitLab CI Process

Configure Secret Variables

In Project → Settings → CI/CD → Variables, create

SONAR_QUBE_TOKEN

,

HARBOR_PASSWORD

, and

CI_PASSWORD

as unprotected hidden variables.

Template Library Resource Update

The template library is described in the linked blog. The

kustomize.yaml

file is used to generate environment‑specific manifests and replace image references before committing back to GitLab.

<code># Update kustomize
variables:
  KUSTOMIZE_OVERLAY: ''
.update-kustomize:
  stage: update-kustomize
  tags:
    - build
  only:
    - master
    - test
  before_script:
    - git remote set-url origin http://${CI_USER}:${CI_PASSWORD}@gitlab.local.com/devops/spring_boot_demo.git
    - git config --global user.email "${CI_EMAIL}"
    - git config --global user.name "${CI_USER}"
    - if [ "$CI_COMMIT_BRANCH" == "master" ]; then KUSTOMIZE_OVERLAY="prod"; fi
    - if [ "$CI_COMMIT_BRANCH" == "test" ]; then KUSTOMIZE_OVERLAY="test"; fi
  script:
    - git checkout -B ${CI_COMMIT_BRANCH}
    - cd cicd/kustomize/overlays/${KUSTOMIZE_OVERLAY}
    - kustomize edit set image $CONTAINER_NAME=$IMAGE_FULL_NAME
    - kustomize build .
    - git commit -am '[gitlab ci] kustomize update'
    - git push origin ${CI_COMMIT_BRANCH}
</code>

Pipeline Configuration

Create a

.gitlab-ci.yml

file at the project root with the following content:

<code>include:
  - project: 'devops/gitlabci-template'
    ref: master
    file: 'jobs/build.yml'
  - project: 'devops/gitlabci-template'
    ref: master
    file: 'jobs/test.yml'
  - project: 'devops/gitlabci-template'
    ref: master
    file: 'jobs/sonarqube.yml'
  - project: 'devops/gitlabci-template'
    ref: master
    file: 'jobs/harbor.yml'
variables:
  SONAR_QUBE_PATH: "$CI_PROJECT_DIR/cicd/sonar-project.properties"
  HARBOR_REPO: devops
  HARBOR_USER: admin
  DOCKERFILE_PATH: cicd/Dockerfile
  IMAGE_NAME: "$CI_PROJECT_NAME:$CI_COMMIT_BRANCH-$CI_COMMIT_SHORT_SHA"
  CI_USER: gitlabci
  CI_EMAIL: [email protected]
  ROLLOUT_PATH: cicd/argo-cd/bluegreen/rollout.yaml
workflow:
  rules:
    - if: '$GITLAB_USER_LOGIN == "gitlabci"'
      when: never
    - when: always
default:
  cache:
    paths:
      - target/
stages:
  - build
  - code_scan
  - product
  - update_yaml
mvn:
  stage: build
  extends: .mvn_build
  image: harbor.local.com/cicd/maven:v3.9.3
  tags:
    - k8s
unit_test:
  stage: build
  extends: .mvn_unit_test
  image: harbor.local.com/cicd/maven:v3.9.3
  tags:
    - k8s
code_scan:
  stage: code_scan
  extends: .sonarqube
  image: harbor.local.com/cicd/sonar-scanner-cli:10
  tags:
    - k8s
product:
  stage: product
  image: harbor.local.com/cicd/gitlab-agent:v1.1
  extends: .container_upload_harbor
  tags:
    - k8s
update_yaml:
  stage: update_yaml
  image: harbor.local.com/cicd/gitlab-agent:v1.1
  extends: .update_kustomize
  tags:
    - k8s
</code>

GitLab CI Result Verification

After the

update_yaml

stage, the generated manifests show updated image tags and namespaces. Screenshots illustrate the updated

kustomization.yaml

and the committed changes.

Argo CD Process (Rolling Update)

Create Application

Create an Argo CD Application via UI, CLI, or YAML. Example Kustomize‑type Application manifest:

<code>apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-prod
  namespace: argocd
spec:
  destination:
    namespace: prod
    server: 'https://kubernetes.default.svc'
  source:
    path: cicd/kustomize/overlays/prod
    repoURL: 'http://gitlab.local.com/devops/spring_boot_demo.git'
    targetRevision: master
  project: devops
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-test
  namespace: argocd
spec:
  destination:
    namespace: test
    server: 'https://kubernetes.default.svc'
  source:
    path: cicd/kustomize/overlays/test
    repoURL: 'http://gitlab.local.com/devops/spring_boot_demo.git'
    targetRevision: test
  project: devops
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
</code>

Argo CD UI shows both applications deployed from

master

and

test

branches.

Result Verification

Check pod status in each namespace:

<code># kubectl get pod -n prod
NAME               READY   STATUS    RESTARTS   AGE
 demo-7dd977b57-5qdcx   1/1   Running   0          4m41s
# kubectl get pod -n test
NAME               READY   STATUS    RESTARTS   AGE
 demo-6b67766cb5-c9fq9   1/1   Running   0          4m32s
</code>

Access the services via host entries to verify the correct version is served.

<code># curl demo.prod.com
<h1>Hello SpringBoot</h1><p>Version:v1 Env:prod</p>
# curl demo.test.com
<h1>Hello SpringBoot</h1><p>Version:v1 Env:test</p>
</code>

After updating the Spring Boot source version from

v1

to

v2

, the CI pipeline rebuilds the image and Argo CD performs a rolling update.

<code># kubectl get pod -n prod
NAME               READY   STATUS        RESTARTS   AGE
 demo-65b44b4d8-58f67   1/1   Running       0          21s
 demo-7dd977b57-5qdcx   1/1   Terminating   0          10m
# curl demo.prod.com
<h1>Hello SpringBoot</h1><p>Version:v2 Env:prod</p>
</code>

Argo CD Process (Blue‑Green Deployment)

Blue‑green deployment relies on the Argo CD Rollouts component. Create a Kustomize‑type Application for the blue‑green rollout.

<code>apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: blue-green
  namespace: argocd
spec:
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    path: cicd/argo-cd/bluegreen
    repoURL: 'http://gitlab.local.com/devops/spring_boot_demo.git'
    targetRevision: master
  project: devops
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
</code>

Verify pods and service responses for both production and test environments, then promote the green version after validation.

<code># kubectl get pod -n prod
NAME               READY   STATUS    RESTARTS   AGE
 bluegreen-rollout-7679f8576-bj9lw   1/1   Running   0          4s
# curl demo.prod.com
<h1>Hello SpringBoot</h1><p>Version:v2 Env:prod</p>
# curl demo.test.com
<h1>Hello SpringBoot</h1><p>Version:v1 Env:prod</p>
# kubectl argo rollouts promote bluegreen-rollout
rollout 'bluegreen-rollout' promoted
</code>

Argo CD Process (Canary Release)

Canary release also uses Rollouts. The only difference is the

ROLLOUT_PATH

points to the canary manifest.

<code>variables:
  ROLLOUT_PATH: cicd/argo-cd/canary/rollout.yaml
</code>

After deploying the canary Application, initial traffic is split (10% canary, 90% stable). Verify by curling the service repeatedly.

<code># for i in {1..10}; do curl canary.demo.com; done
<h1>Hello SpringBoot</h1><p>Version:v3 Env:prod</p>
... (mostly v3, occasional v4 after promotion)
</code>

Promote the canary rollout once the new version is validated:

<code># kubectl argo rollouts promote canary-rollout
rollout 'canary-rollout' promoted
</code>

Argo CD Process (Multi‑Cluster Deployment)

Configure multiple clusters in Argo CD (e.g., a production HA cluster and a test cluster). Add the test cluster via

argocd cluster add

and grant the devops project permission to manage it.

<code># argocd login argocd.local.com
# argocd cluster add test-admin@k8s-test --kubeconfig=/root/.kube/config
</code>

Create two Applications: one targeting the production cluster (master branch) and another targeting the test cluster (test branch).

<code># Production Application (master → prod cluster)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-prod
  namespace: argocd
spec:
  destination:
    namespace: prod
    server: 'https://kubernetes.default.svc'
  source:
    path: cicd/kustomize/overlays/prod
    repoURL: 'http://gitlab.local.com/devops/spring_boot_demo.git'
    targetRevision: master
  project: devops
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
---
# Test Application (test → test cluster)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-test
  namespace: argocd
spec:
  destination:
    namespace: test
    server: 'https://192.168.10.10:6443'
  source:
    path: cicd/kustomize/overlays/test
    repoURL: 'http://gitlab.local.com/devops/spring_boot_demo.git'
    targetRevision: test
  project: devops
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
</code>

Argo CD automatically deploys the applications to their respective clusters. Verify pods in each cluster:

<code># Switch to test cluster
kubectl config use-context test-admin@k8s-test
kubectl get pod -n test
NAME               READY   STATUS    RESTARTS   AGE
 demo-6c86b77bd6-dpf4m   1/1   Running   0          3m3s
# Switch back to production cluster
kubectl config use-context ha-admin@k8s-ha
kubectl get pod -n prod
NAME               READY   STATUS    RESTARTS   AGE
 demo-77b7f4576b-vlwtc   1/1   Running   0          3m
</code>

The guide demonstrates a full end‑to‑end GitOps workflow covering CI pipeline setup, image building, Kustomize manifest management, and automated multi‑environment deployments with rolling, blue‑green, and canary strategies using Argo CD.