Analysis of OpenAI's December 2024 Outage: Kubernetes Control Plane Overload and Mitigation

On December 11, 2024, OpenAI experienced a global severe outage affecting all services including ChatGPT, API, and Sora, with periods of inaccessibility.

The official incident report is available at: https://status.openai.com/incidents/ctrsv3lwd797 .

In this article, I will take you through a deep understanding of OpenAI's incident and draw lessons from others' mistakes.

The main timeline of the incident is as follows:

At 3:17 PM PST on Dec 11, 2024, all OpenAI services began to be unavailable. Over time the problem worsened, causing customers to be unable to access the API, ChatGPT, and Sora at multiple intervals.

At 3:53 PM PST, engineers discovered that API calls returned errors and users could not log into the OpenAI platform, and the issue quickly spread to multiple services.

As time progressed, the OpenAI engineering team identified the root cause and initiated an emergency response. By 7:38 PM PST, all services were fully restored.

From the incident start to full recovery, the duration exceeded four hours, which is a huge impact for globally used services like ChatGPT and Sora.

The root cause was a newly deployed monitoring service that collected metrics from the Kubernetes control plane. Due to a configuration issue, each node in the cluster sent a massive number of requests to the Kubernetes API, overwhelming the control plane and causing it to crash.

The Kubernetes control plane is the core of cluster management, responsible for scheduling, monitoring, and managing cluster state.

Although the Kubernetes data plane (the Nodes and Pods) can continue to run when the control plane is down, this incident revealed a critical dependency—CoreDNS.

CoreDNS is the core component in Kubernetes responsible for service discovery and DNS resolution. It runs in the control plane to provide DNS services for Pods. When the control plane crashed, CoreDNS could not provide effective DNS resolution, causing Pods to be unable to locate other services via DNS.

This issue ultimately triggered a chain reaction, directly causing many services to become unusable.

To address the control plane overload, OpenAI engineers employed several strategies, including:

Scaling down the cluster: Reducing load on the Kubernetes API alleviated pressure on the control plane.

Blocking access to the Kubernetes management API: Prevented new high‑load requests, giving the API server time to recover.

Expanding resources of the Kubernetes API server: Added capacity to handle pending requests until the problem was resolved.

Through these measures, OpenAI successfully restored services and gradually redirected traffic to healthy clusters.

This incident shows that even trillion‑dollar companies like OpenAI have significant gaps in core infrastructure. The control plane lacked sufficient redundant nodes, which is surprising.

I do not want to criticize the team; I trust they have excellent engineers. However, many technical problems arise beyond engineers' control, especially when companies pursue cost‑cutting and efficiency, putting pressure on infrastructure teams.

Cost‑driven decision making can lead teams to prioritize short‑term savings over long‑term risk, and cutting infrastructure can cause disastrous outcomes.

This year, several major incidents at large firms were caused by infrastructure problems. For businesses heavily dependent on the internet, such impacts directly affect core service stability and sustainability.

Therefore, I recommend that leading companies invest adequately in infrastructure; not all areas are suitable for cost‑cutting to improve efficiency. Over‑reducing spending on technology and infrastructure can create unforeseen risks.

At the end, I recommend my tutorial: a Kubernetes system tutorial designed for enterprise practice, based on my years of experience, tailored for readers who want to master Kubernetes.

Interested readers can subscribe.

------------------ END ------------------

Follow the public account to get more exciting content