Strategies for Rolling Restart of Pods During Istio Service Mesh Upgrade
To upgrade an Istio service mesh without overloading the cluster or causing downtime, the author recommends using Kubernetes’s built‑in kubectl rollout restart for each deployment—scaling replicas up then deleting old pods or simply invoking the command in a scripted loop—to safely perform a rolling restart of all sidecar‑proxied pods.
When upgrading Istio service mesh, all pods must be restarted to update the sidecar proxy to the new version. A naive approach of deleting all pods in a namespace causes high scheduling pressure on Kubernetes and can disrupt services, especially for deployments with a single replica.
An initial improvement involved iterating over deployments, deleting one pod at a time with a five‑minute sleep, but this still failed when a deployment had only one pod or when pods failed to restart due to missing images or init‑container errors.
A more robust method scales the deployment replica count up by one, waits for the new pod to become running, then deletes the old pod, and finally scales the replica count back down. This ensures zero downtime but requires complex orchestration.
The author discovered that kubectl rollout restart deployment/<name> performs a rolling restart with the same semantics, handled by the Kubernetes control plane. This command can be used in scripts to restart each deployment safely.
Since the Kubernetes client‑go SDK does not expose a rollout restart API, the author reused the kubectl source code, adding a dependency on k8s.io/kubectl and adapting the RestartOptions structure to invoke the restart programmatically.
Finally, a simple loop over all deployments in a namespace calls the rollout restart function with a short sleep between iterations, providing a reliable and low‑impact way to update Istio sidecars.
Sohu Tech Products
A knowledge-sharing platform for Sohu's technology products. As a leading Chinese internet brand with media, video, search, and gaming services and over 700 million users, Sohu continuously drives tech innovation and practice. We’ll share practical insights and tech news here.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.