Azure Leap‑Year Outage and Leap‑Second Impacts on Cloud Systems

On February 29, 2012, Microsoft Azure suffered a severe outage where some hosts were down for up to eight hours because a leap‑year date was handled incorrectly.

You Really Understand Leap Years and Leap Months?

Azure Outage Review

Azure 's physical architecture consists of clusters of about 1,000 physical hosts, each managed by a Fabric Controller ( FC ) that handles host lifecycle and automatic operations. The design limits failures to affect only a single cluster rather than the entire Azure service.

Within each cluster, every virtual machine (VM) runs a Guest Agent ( GA ) that communicates with the Host Agent ( HA ) on the physical host. During VM startup, the GA generates a public key and sends it to the HA . Subsequent certificates (e.g., HTTPS , SSL ) are encrypted with this key and sent back to the GA .

How Did the Failure Occur?

For safety, the GA sent a public‑key expiration time set one year ahead using the following code:

SYSTEMTIME st; // declare a SYSTEMTIME variable
GetSystemTime(&st); // set it to the current date and time
st.wYear++; // increment it by one year

If the current date was February 29, 2012, the code produced an expiration date of February 29, 2013, which does not exist. The VM failed to start, and after three unsuccessful attempts the HA reported a hardware fault to the FC , triggering a self‑recovery process that replaced the host. Because the failure was reported at the cluster level, the entire cluster became unavailable.

Leap‑Year Edge Cases to Watch

Common pitfalls include:

Adding 365 days to the last day before a leap year does not land on the last day of the next year.

Adding 365 days to the first day of a leap year lands on the last day of that year, not the first day of the following year.

Using a fixed array size of 365 for day‑of‑year indexing causes out‑of‑bounds errors on leap‑year day 366.

February in a leap year has 29 days; adding 28 days to February 29 yields March 1 of the next year.

Have You Heard of Leap Seconds?

Three time standards are relevant:

UT (Universal Time) – based on Earth's rotation, which varies due to tidal forces, mantle movements, etc., causing daily errors of several milliseconds.

TAI (International Atomic Time) – defined by atomic clocks with nanosecond accuracy, maintained by the BIPM.

UTC (Coordinated Universal Time) – aligns TAI with UT by inserting or removing leap seconds when the difference exceeds 0.9 seconds.

Leap seconds are typically added at the end or middle of a year, announced six months in advance by the IERS.

What Happens When Time Goes Backward?

In 2017, Cloudflare's DNS service experienced a failure because a leap second caused the system clock to repeat a second, making the round‑trip time calculation rtt := time.Now().Sub(start) negative. The negative RTT propagated through the RRDNS load balancer, leading to a panic in rand.Int63n and a cascade of DNS resolution failures.

// Update upstream sRTT on UDP queries, penalize it if it fails
if !start.IsZero() {
  // note: start is the request start time
  rtt := time.Now().Sub(start)
  if success && rcode != dns.RcodeServerFailure {
    s.updateRTT(rtt)
  } else {
    // The penalty should be a multiple of actual timeout
    s.updateRTT(TimeoutPenalty * s.timeout)
  }
}

Does Java Have Similar Issues?

The java.util.Date class claims to represent UTC but may not do so accurately on all JVMs. On POSIX‑compliant hosts that disallow leap seconds, the JVM cannot represent the extra second, leading to time‑drift problems similar to those seen in other languages.

To mitigate, many companies manually adjust NTP servers during a leap second, spreading the extra second over the whole day. However, if some clusters use adjusted NTP while others use standard NTP, time mismatches can still occur, causing issues in time‑sensitive applications such as financial settlement.