Analyzing Sentinel's Slow Call Ratio Circuit Breaker

With the popularity of micro‑services, the stability between services becomes increasingly important. Sentinel is a flow‑control component for distributed service architectures that helps ensure stability through traffic control, circuit breaking, and adaptive system protection.

During a recent online stability inspection, an occasional circuit‑break of the Ab interface was observed. The configured Sentinel rule triggers a circuit break when the slow‑call ratio exceeds 50% within a 1‑second statistical window.

Log analysis showed that the number of slow calls was far below the configured threshold, prompting a deep dive into Sentinel’s source code.

Key conclusion: Sentinel evaluates the slow‑call ratio on every request, not only after the sliding window ends, which can cause early circuit breaking.

In the project, resources are protected with the @SentinelResource annotation. When a rule is violated, the specified blockHandler method is invoked.

@SentinelResource(value = "pandora.abService.callAb", blockHandler = "callAb", blockHandlerClass = SentinelFallbackHandler.class)
public String callAb(Long userId, String key) {
    // ...
}

The SentinelResourceAspect bean processes these annotations:

@Bean
@ConditionalOnMissingBean
public SentinelResourceAspect sentinelResourceAspect() {
    return new SentinelResourceAspect();
}

Request entry follows the chain: SphU.entry() → Sph.entry() → CtSph.entry() , which eventually calls entryWithPriority() to handle the resource.

private Entry entryWithPriority(ResourceWrapper resourceWrapper, int count, boolean prioritized, Object... args) throws BlockException {
    // get context
    Context context = ContextUtil.getContext();
    // get processing chain
    ProcessorSlot
chain = lookProcessChain(resourceWrapper);
    new CtEntry(resourceWrapper, chain, context);
    try {
        // chain processing
        chain.entry(context, resourceWrapper, null, count, prioritized, args);
    } catch (BlockException e1) {
        e.exit(count, args);
        throw e1;
    } catch (Throwable e1) {
        RecordLog.info("Sentinel unexpected exception", e1);
    }
    return e;
}

Sentinel maintains two time‑window levels (second‑level and minute‑level) using LeapArray and MetricBucket . Each MetricBucket records counters for events such as PASS, BLOCK, EXCEPTION, SUCCESS, RT, and OCCUPIED_PASS.

private final LongAdder[] counters;
public MetricBucket add(MetricEvent event, long n) {
    counters[event.ordinal()].add(n);
    return this;
}
public enum MetricEvent { PASS, BLOCK, EXCEPTION, SUCCESS, RT, OCCUPIED_PASS }

The current window is obtained via LeapArray.currentWindow() , which calculates the index based on the current timestamp and creates or resets windows as needed.

public WindowWrap
currentWindow() {
    return currentWindow(TimeUtil.currentTimeMillis());
}
public WindowWrap
currentWindow(long timeMillis) {
    int idx = calculateTimeIdx(timeMillis);
    long windowStart = calculateWindowStart(timeMillis);
    while (true) {
        WindowWrap
old = array.get(idx);
        if (old == null) {
            WindowWrap
window = new WindowWrap<>(windowLengthInMs, windowStart, newEmptyBucket(timeMillis));
            if (array.compareAndSet(idx, null, window)) {
                return window;
            } else {
                Thread.yield();
            }
        } else if (windowStart == old.windowStart()) {
            return old;
        } else if (windowStart > old.windowStart()) {
            if (updateLock.tryLock()) {
                try { return resetWindowTo(old, windowStart); }
                finally { updateLock.unlock(); }
            } else {
                Thread.yield();
            }
        } else {
            return new WindowWrap<>(windowLengthInMs, windowStart, newEmptyBucket(timeMillis));
        }
    }
}

StatisticSlot records runtime metrics. Its entry() method first invokes downstream slots (SystemSlot, FlowSlot, DegradeSlot, etc.). If a rule blocks the request, a BlockException is thrown and the block count is recorded; otherwise, pass counts and thread numbers are updated.

@Override
public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args) throws Throwable {
    try {
        fireEntry(context, resourceWrapper, node, count, prioritized, args);
        node.increaseThreadNum();
        node.addPassRequest(count);
    } catch (BlockException e) {
        context.getCurEntry().setBlockError(e);
        node.increaseBlockQps(count);
        if (context.getCurEntry().getOriginNode() != null) {
            context.getCurEntry().getOriginNode().increaseBlockQps(count);
        }
        throw e;
    } catch (Throwable e) {
        context.getCurEntry().setError(e);
        throw e;
    }
}

The pass‑request update simply adds counts to second‑ and minute‑level rolling counters:

@Override
public void addPassRequest(int count) {
    rollingCounterInSecond.addPass(count);
    rollingCounterInMinute.addPass(count);
}

Sentinel’s sliding windows for second‑level metrics consist of two 500 ms sub‑windows (default SampleCountProperty.SAMPLE_COUNT = 2 ).

// SampleCountProperty.SAMPLE_COUNT defaults to 2
private volatile Metric rollingCounterInSecond = new ArrayMetric(SampleCountProperty.SAMPLE_COUNT, IntervalProperty.INTERVAL);
public ArrayMetric(int sampleCount, int intervalInMs) {
    this.data = new OccupiableBucketLeapArray(sampleCount, intervalInMs);
}
public LeapArray(int sampleCount, int intervalInMs) {
    this.windowLengthInMs = intervalInMs / sampleCount;
    this.array = new AtomicReferenceArray<>(sampleCount);
}

The circuit‑breaker logic resides in DegradeSlot . When a slow‑call ratio rule is configured, ResponseTimeCircuitBreaker tracks slow calls in a single sliding window (e.g., 30 s). In exit() , it records the request’s response time and updates counters.

public void exit(Context context, ResourceWrapper r, int count, Object... args) {
    Entry curEntry = context.getCurEntry();
    if (curEntry.getBlockError() == null) {
        for (CircuitBreaker circuitBreaker : circuitBreakers) {
            circuitBreaker.onRequestComplete(context);
        }
    }
    fireExit(context, r, count, args);
}
@Override
public void onRequestComplete(Context context) {
    SlowRequestCounter counter = slidingCounter.currentWindow().value();
    Entry entry = context.getCurEntry();
    long rt = Math.max(0, System.currentTimeMillis() - entry.getCreateTimestamp());
    if (rt > maxAllowedRt) {
        counter.slowCount.add(1);
    }
    counter.totalCount.add(1);
    handleStateChangeWhenThresholdExceeded(rt);
}

The breaker transitions:

If the current window’s total calls are below the minimum (e.g., 5), no check is performed.

When total calls ≥ minimum, the slow‑call ratio is computed. If it exceeds the configured threshold (e.g., 50 %), the breaker moves from CLOSED to OPEN and sets a retry timestamp (e.g., 3 s later).

After the timeout, the breaker enters HALF‑OPEN, allowing one probe request. A normal response closes the breaker; a slow response re‑opens it.

This explains why the Ab interface occasionally trips the circuit breaker: the first few requests in a new window can push the slow‑call ratio over the threshold, causing an immediate break before the window fully accumulates data.

In summary, Sentinel’s slow‑call ratio circuit‑breaker evaluates each request against a sliding window, uses a chain‑of‑responsibility pattern for rule processing, and manages state transitions (CLOSED → OPEN → HALF‑OPEN → CLOSED) based on real‑time statistics.