Understanding the Internal Process of kubectl Commands (e.g., kubectl exec)

kubectl is the command‑line tool for interacting with a Kubernetes cluster, allowing users to operate and manage cluster resources. Have you ever wondered what actually happens behind the scenes after you run a kubectl command?

Detailed Process

kubectl → kube‑api‑server

Depending on the communication type, kubectl commands can be classified into two categories: one‑way communication and two‑way communication.

• One‑way communication: kubectl sends a request to the kube‑api‑server over HTTPS and receives a response. Most kubectl commands—such as creating, deleting, updating, and querying resources—are one‑way.

• Two‑way communication: For a few persistent‑operation commands like exec , attach , port‑forward , and logs , kubectl and the kube‑api‑server upgrade the connection to a WebSocket , enabling continuous bidirectional message exchange.

For most one‑way commands, after the kube‑api‑server processes the request (potentially interacting with etcd) and sends a response back to kubectl , the interaction ends, even though other cluster components may continue internal processing.

In contrast, two‑way commands involve a longer processing chain. The following example uses kubectl exec -it nginx -- bash to illustrate the detailed flow.

As shown in the diagram, kubectl first sends an HTTPS request to the kube‑api‑server , negotiates an upgrade to a WebSocket , and then maintains a persistent bidirectional channel with the server.

kube‑api‑server → kubelet

The kubelet starts three servers:

• HTTPS server : the main interface for the kubelet to interact with the kube‑api‑server and other components, handling health checks, metrics, pod management, and APIs such as /exec , /attach , /portForward , etc.

• HTTP Read‑Only server : provides read‑only API endpoints; it is disabled by default for security and can be enabled with the --read-only-port flag (not recommended in production).

• gRPC server : used to query resource allocation of Pods and containers on the node.

/exec/{podNamespace}/{podID}/{containerName}
/attach/{podNamespace}/{podID}/{containerName}
/portForward/{podNamespace}/{podID}/{containerName}
/containerLogs/{podNamespace}/{podID}/{containerName}

When the kube‑api‑server receives the kubectl exec request, it forwards the request to the appropriate kubelet via HTTPS, and the kubelet continues the processing.

kubelet → CRI

The kubelet uses gRPC to call the CRI (Container Runtime Interface) component, passing the execution instruction to the container runtime.

The gRPC protocol defines two services— RuntimeService and ImageService —with several methods, including:

RuntimeServer.ExecSync

RuntimeServer.Exec

RuntimeServer.Attach

RuntimeServer.PortForward

The CRI component finally executes the command and streams the result (stdout/stderr) back through the kubelet and kube‑api‑server to the original kubectl client, where it is displayed.

Summary

The text describes the complete lifecycle of a kubectl command, using kubectl exec as an example, and shows how kube‑api‑server , kubelet , and the CRI collaborate via various protocols such as HTTPS , WebSocket , and gRPC .

(Follow me for ad‑free technical content; no sensationalism.)

References:

https://kubernetes.io/docs/reference/kubectl/generated/

https://erkanerol.github.io/post/how-kubectl-exec-works/

https://github.com/kubernetes/kubernetes/blob/v1.31.1/pkg/kubelet/server/server.go