Implementing HTTP Request Retry, Hedging, and Circuit Breaking in Go

In unreliable network environments, reliable communication often requires HTTP request retry mechanisms. The Go standard library net/http does not provide built‑in retry, so this article explains how to implement retry, backoff strategies, hedged requests, and circuit breaking.

1. Overview

Perceive Errors : Identify error types via HTTP status codes (e.g., 4xx client errors should not be retried).

Retry Decision : Decide whether to retry based on error type and business tolerance.

Retry Strategy : Define interval, count, and backoff (linear, linear‑jitter, exponential, exponential‑jitter).

Hedging Strategy : Send multiple parallel requests without waiting for the first response and use the earliest successful reply.

Circuit Breaker : Prevent resource exhaustion by opening the circuit after a threshold of failures or high latency.

2. Retry Strategies

The trade‑off between request latency tolerance and downstream load leads to several backoff rules:

Linear Backoff – fixed interval (e.g., 1 s).

Linear Jitter – fixed interval plus a random jitter to avoid thundering herd.

Exponential Backoff – interval grows exponentially (e.g., 3 s, 9 s, 27 s).

Exponential Jitter – exponential interval with added random jitter.

Adding jitter helps prevent many clients from retrying simultaneously.

3. Using net/http Directly

When retrying POST requests, the request body must be reset because the io.Reader is consumed after the first attempt. Example:

req, _ := http.NewRequest("POST", "localhost", strings.NewReader("hello"))

If the server reads the body and the client retries without resetting, the body length becomes zero, causing errors.

Resetting the body:

func resetBody(request *http.Request, originalBody []byte) {
    request.Body = io.NopCloser(bytes.NewBuffer(originalBody))
    request.GetBody = func() (io.ReadCloser, error) {
        return io.NopCloser(bytes.NewBuffer(originalBody)), nil
    }
}

Full retry implementation with status‑code check, backoff, and body reset:

func retryDo(req *http.Request, maxRetries int, timeout time.Duration, backoffStrategy BackoffStrategy) (*http.Response, error) {
    var originalBody []byte
    if req != nil && req.Body != nil {
        originalBody, _ = copyBody(req.Body)
        resetBody(req, originalBody)
    }
    // ... loop with client.Do, status code check, resetBody, sleep(backoffStrategy(i)) ...
}

4. Hedged Requests

Hedging sends several concurrent requests and returns the first successful response. It requires cloning the request (including context) and resetting the body for each clone:

req2 := req.Clone(req.Context())
contents, _ := io.ReadAll(req.Body)
contents2, _ := io.ReadAll(req2.Body) // second read is empty without reset

Implementation uses goroutines, sync.WaitGroup , and channels to control concurrency and collect the first successful result:

func retryHedged(req *http.Request, maxRetries int, timeout time.Duration, backoffStrategy BackoffStrategy) (*http.Response, error) {
    // copy original body, create copyRequest closure, launch goroutines per attempt,
    // send successful response on multiplexCh, wait for all failures on allRequestsBackCh.
}

5. Circuit Breaking & Degradation

To avoid cascading failures, a circuit breaker monitors request count, error ratio, and average response time. When thresholds are exceeded, the breaker opens, rejecting further calls until a timeout expires and the circuit half‑opens for probing. Common strategies:

Error‑ratio based: open when failure rate exceeds a configured percentage.

Average‑RT based: open when average latency exceeds a threshold.

Example using hystrix-go :

hystrix.ConfigureCommand("my_service", hystrix.CommandConfig{ErrorPercentThreshold: 30})
hystrix.Do("my_service", func() error {
    req, _ := http.NewRequest("POST", "http://localhost:8090/", strings.NewReader("test"))
    _, err := retryDo(req, 5, 20*time.Millisecond, ExponentialBackoff)
    return err
}, func(err error) error { return nil })

6. Summary

The article walks through the necessity of retry mechanisms for HTTP calls in Go, presents concrete code for resetting request bodies, demonstrates backoff policies, shows how to implement hedged requests with concurrency control, and finally integrates circuit breaking to protect services from overload and cascading failures.