Lessons Learned from Migrating Applications to Kubernetes

1. Reasons for Migrating to Kubernetes

We abandoned Ansible‑based deployments on EC2 about two years ago in favor of containerization and the Kubernetes stack for application orchestration, eventually moving most of our infrastructure to Kubernetes despite the technical challenges of running a hybrid environment and retraining the team.

If you are starting a new business, using containers and Kubernetes is advisable provided you have sufficient bandwidth and the expertise to configure and operate it; even on managed services like EKS, GKE, or AKS there is a learning curve and a DevOps culture is required.

Consider whether you truly need to migrate existing workloads to Kubernetes; the migration is arduous and should be motivated by clear goals such as establishing a continuous‑integration infrastructure to accelerate microservice refactoring, which was our primary driver.

2. Migration Approach

Stabilizing a complex CI pipeline on Kubernetes took about a year and a half, during which we built additional tools, telemetry, and re‑engineered deployment methods to keep development and production environments consistent.

We previously used Ansible, HashiCorp Consul, and Vault for infrastructure and configuration; while they worked, we realized that Kubernetes could provide service discovery, cost management, elasticity, and governance, albeit with added operational overhead.

3. Out‑of‑the‑Box Kubernetes Is Not Sufficient

Kubernetes is a platform for building PaaS solutions rather than a turnkey PaaS itself; most teams need to add components such as metrics, logging, service discovery, distributed tracing, config and secret management, CI/CD, and local development tooling to create an internal platform.

Metrics: We chose Prometheus, moving away from InfluxDB.

Logging: After evaluating ELK, we adopted Grafana Loki for its simplicity and PromQL‑compatible query language.

Config and Secret Management: We use Consul, Vault, and Consul templates as init containers to refresh secrets and reload applications.

CI/CD: We continued using Jenkins but are exploring Tekton and Argo Workflows, alongside tools like Skaffold and Telepresence for deployment.

4. Operating a Kubernetes Cluster Is Hard

We initially built our own cluster with kops in AWS Singapore because EKS was unavailable; configuring autoscaling, resources, and networking proved non‑trivial, and most default settings do not work well in production.

We recommend offloading operational complexity to managed services when possible and adopting GitOps practices with tools like eksctl, Terraform, and automated pipelines.

Resource requests and limits must be balanced to avoid pod eviction while preventing waste; we learned to keep requests high enough for stability but not excessive, and limits close to requests to allow headroom.

5. Security, Governance, and Cost

We use Open Policy Agent to enforce security policies such as preventing public ELBs without proper annotations.

Cost savings were significant after migration, though we initially over‑provisioned resources; using spot instances and careful instance type selection further reduced expenses.

We integrated Ingress controllers to replace LoadBalancer services with NodePort + Ingress, lowering ELB costs, and are investigating service mesh solutions to control cross‑AZ traffic.

6. Extending Kubernetes with CRDs, Operators, and Controllers

We developed custom controllers and CRDs to automate tasks such as converting LoadBalancer services to Ingress, creating DNS CNAME records, and generating Grafana dashboards declaratively, simplifying daily operations for developers.