1 views collected around this technical thread.
This article explains the differences between SLA, SLO, and SLI, shows how to express user expectations as concrete service level agreements, and introduces essential high‑availability metrics such as availability percentages, MTBF, MTTR, RPO, RTO, WRT, and MTD for reliable system design.
This article explains what performance testing is, why it is essential for reliable systems, outlines practical steps for conducting effective tests, and introduces the wrk benchmarking tool as a lightweight solution for generating realistic load and measuring key performance metrics.
The article explains how performance testing measures system efficiency under load while fault testing validates resilience under abnormal conditions, highlighting their shared goals, differences, overlapping toolchains, and how their combined use drives architecture optimization and improves service level agreements in modern complex software systems.
This article outlines fifteen common operational and management mistakes—such as frequent incidents, excessive new hires, lack of automation, and missing rollback plans—that can trigger system outages, and offers guidance on how teams can strengthen testing, processes, and team capabilities to prevent downtime.
Bilibili’s engineering team built a scenario‑metadata and one‑click fault‑drill platform, implemented multi‑tier degradation, dynamic capacity planning, and extensive automated fault‑injection testing to guarantee zero‑severity incidents during the high‑traffic 2025 Spring Festival Gala live stream.
To prevent performance degradation as systems scale, the team implemented an automated load‑testing and circuit‑breaker workflow that runs in the release pipeline, compares real‑time metrics against a baseline of CPU, QPS, memory and latency, blocks releases exceeding a 10 % drop, and logs issues, resulting in thousands of tests, dozens of bugs fixed, and up to 90 % faster wordlist creation.
This article explains common service problems and presents practical solutions such as rate limiting, circuit breaking, and degradation, detailing their principles, implementation methods—including Nginx, token‑bucket, sliding‑window algorithms, and Go‑zero code examples—while highlighting key considerations for building resilient microservice systems.
Amid a surge of high‑profile outages, the CAICT introduces a comprehensive SRE framework that addresses large‑scale, high‑frequency changes, complex tech stacks, and massive traffic, outlining development and operational reliability practices, maturity levels, and actionable guidelines to enhance system stability.
Understanding system health is essential for reliable services, and this guide explains how to use powerful monitoring tools to collect, visualize, and alert on the four golden signals—latency, traffic, errors, and saturation—across servers, applications, and external dependencies, helping teams detect and resolve issues efficiently.
The article explores the growing importance of system reliability in China, the national regulations driving SRE adoption, Inner Mongolia Mobile’s successful Level‑3 SRE assessment at the 2024 GOPS conference, and insights from Deputy GM Zhang Yongtao on practices, challenges, and future plans.
The team built a cross‑platform real‑time troubleshooting system for live streaming that adds critical‑business monitoring and a unified trace_id‑based tracing framework, simplifies OpenTracing, iterates reporting components, handles multi‑threading, stitches telemetry into searchable event chains, and via dashboards cut diagnosis time from two hours to five minutes, achieving a 91% fault‑resolution rate.
In the digital era, a capable IT operations team ensures system stability, reduces costs, accelerates issue resolution, strengthens security, supports product development, and improves user experience, making it a critical driver of overall business value.
The article explains fault isolation as a key technique for improving distributed system availability, detailing multiple isolation levels—from data‑center to user‑level—and complementary strategies such as circuit breakers, timeouts, fast‑fail, load balancing, caching, and degradation switches.
This article shares ten practical operations strategies—ranging from change‑rollback procedures and cautious handling of destructive commands to robust backup verification, alerting, and meticulous hand‑over practices—that together help teams dramatically reduce system outages and maintain high availability.
The article explains what interface‑level failures are, why they occur due to internal bugs or external overload, and presents four practical mitigation techniques—degradation, circuit breaking, rate limiting, and queuing—detailing their principles, implementation options, and trade‑offs for reliable system operation.
The 1996 Ariane 5 launch failed and exploded due to a single line of legacy code that caused a 64‑bit floating‑point to 16‑bit signed integer conversion overflow in the guidance system, highlighting the dangers of unchecked code reuse, inadequate error handling, and insufficient testing in critical software.
This article details HuoLala's three‑year journey of systematic business‑peak assurance, covering goal definition, project‑management practices, technical risk mitigation, cloud‑provider coordination, and post‑event reviews that together delivered zero‑fault high‑traffic periods and continuously improving system stability.
This guide explains how to use the four golden signals—latency, traffic, errors, and saturation—to design effective monitoring across servers, services, and external dependencies, helping teams detect issues early and maintain reliable, high‑performance systems.
The article outlines a three‑stage evolution of operations—from rapid incident response to proactive fault‑injection—while offering practical guidance on improving availability, visualizing changes, and aligning technical metrics with business value to elevate the role of operations engineers.
iQIYI implemented full‑link load testing of its playback service using LoadMaker for traffic generation and Rover for link control, mapping the topology, creating weighted user scenarios, and safely pressurizing production‑like environments, which validated multi‑times historical peak capacity, uncovered bottlenecks, and enabled several performance and disaster‑recovery improvements without impacting real users.