This article shares concrete SRE‑inspired techniques—duty‑roster scheduling, tiered alert handling, automation safeguards, dashboard focus on top‑3 alerts, time‑based filtering, and systematic code review—to dramatically reduce daily alarm volume while keeping on‑call teams motivated and effective.
The Open Alarm Platform provides a flexible data model, modular architecture, and robust stability features to enable various business lines to integrate their custom monitoring systems via APIs, offering alert convergence, merging, multi‑channel delivery, and comprehensive management while reducing development and maintenance costs.
The article reflects on handling online incidents by first extinguishing the immediate problem, then digging into root causes, and expands the discussion to what truly constitutes technical ability, the pitfalls of reinventing solutions, raising one’s perspective, and the critical role of systematic retrospection.
The article draws parallels between emergency rooms and IT operations, describing how Kanban's WIP limits, one‑to‑one liaison models, transparent dashboards, and Scrum time‑boxing (daily stand‑ups and weekly reviews) help a globally distributed team prioritize urgent incidents while still advancing important non‑urgent work.
A collection of vivid DBA anecdotes reveals common pitfalls—from missed alerts and accidental production restarts to unsafe terminal habits—and distills practical safeguards that any database operator can adopt to avoid costly mishaps.
A personal account of switching jobs and the first day on an operations team, detailing a critical menu‑click error, the rapid response, and the deeper reflections on risk awareness, teamwork, and professional growth in high‑stakes production environments.
This article outlines Youzan's end‑to‑end online incident management process—from fault detection and coordination through root‑cause analysis, recovery, review, and actionable JIRA tracking—highlighting practical workflows, data analysis, and continuous improvement practices for reliable service delivery.
This talk explores how ops engineers can achieve continuous professional development by adopting checklist thinking, covering growth drivers, error classification, practical checklist applications, cognitive models, and design principles that turn complex incidents into systematic, repeatable processes.
This article examines the growing challenges of system availability in modern operations, explains the concept of availability and the N‑nine metric, introduces Business Continuity Management and Google SRE approaches, and provides concrete technical and managerial methods—including architecture standardization, scaling strategies, tooling, emergency drills, and incident‑centralized management—to improve operational reliability.
A personal account of switching jobs, the challenges of a first production shift, a critical menu‑click error, the team’s rapid response, and the lasting operational lessons learned about risk awareness, double‑checking, and continuous improvement.
This article shares practical SRE‑based principles for diagnosing and resolving online incidents, emphasizing systematic investigation, gathering clues, and prioritizing service restoration over immediate root‑cause identification to make troubleshooting less mystical and more effective.
The article describes how a ticketing system with strict SLA enforcement, automated routing, and team‑based service management enabled rapid, orderly issue handling during the high‑volume Double 11 shopping event, achieving near‑90% resolution within 30 minutes and improving overall business stability.
This article outlines a systematic methodology for understanding, evaluating, and quickly resolving production system incidents by categorizing system layers, assessing impact, employing Linux diagnostic tools, and designing fault‑tolerant mechanisms to minimize downtime and maintain core functionality.
This article details Ele.me's rapid expansion challenges and shares a three‑stage technical operations journey—fine‑grained division, stability maintenance, and efficiency gains—highlighting real incidents, monitoring upgrades, capacity testing, and practical insights for reliable large‑scale delivery platforms.
This reading note introduces the DevOps three‑step workflow from The Phoenix Project, outlines its three core practices, and summarizes fifteen scenarios illustrating common operational challenges such as untested deployments, missing change records, incident response, and improvement actions.
This article explains how Tencent's social platform operations team prepares for massive holiday traffic spikes by following a four‑stage process—business preparation, capacity evaluation, resource provisioning, and scaling with stress testing—while detailing team structures, operational standards, and the supporting tool ecosystem that enable reliable, high‑availability services.
A sudden rain‑triggered alert cascade exposed a mis‑tested data‑migration tool that overwrote production routing, causing a DB success‑rate drop to zero; the incident narrative details detection, root‑cause tracing, and a set of DevOps tool‑development standards to prevent future outages.
This article shares practical SRE‑based principles and step‑by‑step methods for diagnosing and resolving online incidents, emphasizing mindset, systematic information gathering, and structured analysis to turn mysterious outages into solvable problems.
This article shares practical insights from Google SRE on on‑call duty, covering why on‑call is needed, common challenges, effective scheduling, evaluation methods, and actionable tips to improve team resilience and reduce stress for operations engineers.
A cautionary tale of a rookie who erased a production database on day one, followed by real‑world anecdotes, Reddit and Twitter reactions, and practical advice on backups, ticket handling, and strict operational rules to prevent costly data‑loss incidents.
This article outlines a comprehensive methodology for handling large‑scale online service incidents, covering goals, the "jump‑fill‑avoid" framework, step‑by‑step processes for detection, diagnosis, remediation, and post‑mortem analysis, as well as essential monitoring, logging, and escalation infrastructure.
When a Saturday early-morning outage left the company’s website and mobile app inaccessible for many users, the team traced the issue to an unpaid domain causing DNS resolution failures, detailed the investigation steps, temporary fixes, and lessons learned about DNS processes and operational readiness.
This article shares a detailed case study of how a financial services team restructured European application operations, applied lean retrospectives, built a top‑down monitoring system, and introduced systematic stakeholder collaboration to dramatically improve incident response, system robustness, and user satisfaction.
This article reviews sixteen notable 2016 service outages across finance, cloud, and entertainment, analyzes their causes—ranging from power failures to DDoS attacks—and highlights the critical need for robust disaster‑recovery and information‑security practices.
The article explores personal heroism in operations, defining it as reliance on individual effort to keep flawed systems appearing normal, examines its short‑term benefits and long‑term drawbacks for companies, teams, and the heroes themselves, and offers practical strategies to eliminate this risky mindset.
This article explains Google’s SRE postmortem philosophy, the criteria for writing postmortems, best practices for a blame‑free culture, and how collaborative knowledge‑sharing and incentives improve incident handling and overall system reliability.
This talk details Ctrip's application operations framework, covering data‑center scale, multi‑application deployment on Windows, high availability goals, capacity‑prediction models, disaster‑recovery design, incident response, and the evolution from manual tooling to automated, intelligent operations.
This article recounts two real‑world operations incidents—a network outage caused by an improperly configured portfast on a trunk link and an NFS failure that crippled an API service—then distills practical lessons on pre‑incident procedures, monitoring, fault handling, recovery, and post‑mortem practices.
This article examines recent high‑profile internet outages caused by human error, explores why operations teams are especially prone to mistakes despite automation and standards, and offers practical strategies—such as hiring the right people, fostering safety awareness, and turning professionalism into habit—to reduce future incidents.
This article summarizes a detailed Q&A from the Efficient Operations talk, covering BlueKing’s integration with databases, agent resource management, alarm de‑duplication, automation workflows, development language choices, data handling, and the platform’s suitability for various enterprise environments.
This article explores Randy Shoup’s interview on Google’s DevOps culture, detailing how high‑efficiency organizations instantly detect problems, use swarming to resolve them, document lessons as new knowledge, and foster a blameless post‑mortem culture that drives continuous improvement.
The article examines Ctrip’s prolonged service restoration after a May 28 incident, analyzing the complexities of SOA‑based architectures, the pitfalls of black‑box operations, and how transitioning to white‑box, DevOps‑aligned practices can prevent similar outages.
