This article explains why metrics are essential, defines what a metric is, describes the four essential elements of metric design, outlines the three‑step design process, discusses measurement scales and time characteristics, and provides criteria for evaluating good metrics.
At the 21st GOPS Global Operations Conference in Shanghai, Qunar Travel’s tech expert Zou Sheng shared a detailed hybrid‑cloud container stability practice covering IDC‑first deployment, resource utilization over 60%, phased migration, reliability improvements, AZ monitoring, and cost‑saving strategies.
This article describes Ctrip's optical transport network (TOTN) architecture, analyzes frequent fiber‑cut incidents and resulting device port flapping, presents technical research on fast optical switching and alarm delay, and details an optimization plan that achieved sub‑100 ms fault‑free switchover and stable Redis performance.
GitOps leverages Git's version‑control strengths to automate, codify, and make transparent infrastructure changes, combining IaC, merge requests, and CI/CD, while exploring its principles, toolchains like FluxCD, ArgoCD, Jenkins X, and practical implementations such as SRE Stack for end‑to‑end change management.
The article examines the rapid growth of domestic databases in China’s banking sector, identifies eight critical pain points—from product stability and resource consumption to tooling gaps and migration difficulties—and offers detailed countermeasures covering version upgrade planning, resource optimization, functional testing, skill development, monitoring, ecosystem building, data migration, and backup‑recovery improvements.
This article details the challenges of managing a complex, multi‑layered offline testing environment at KuJiaLe, outlines the standardization of baseline, functional, and integration environments, and explains the comprehensive stability measures—including infrastructure upgrades, automated checks, emergency response, and daily operations—that dramatically improved reliability.
GitHub detailed a year‑long, multi‑team effort to upgrade over 1,200 MySQL hosts from 5.7 to 8.0 using phased rollouts, automated testing, compatibility checks, and rollback mechanisms while maintaining strict SLOs and high‑availability requirements.
This article examines how two‑sided platforms such as Uber, Lyft, DoorDash and Didi design and evaluate peak‑surcharge experiments, addressing network effects, bias‑variance trade‑offs, time‑space slicing, random‑saturation designs, and continuous bandit‑based testing within an operations‑focused experimental system.
This extensive tutorial explains Nginx's architecture, installation, directory layout, configuration directives, location matching rules, reverse proxy setup, load balancing strategies, static‑dynamic separation, CORS handling, caching mechanisms, access control lists, rate limiting, HTTPS configuration, compression, and many other essential directives for effective web server and reverse‑proxy management.
On November 25, Dewu Technology hosted an SRE Stability Engineering salon in Hangzhou where experts from Alibaba, Tencent, Ant Group and Dewu shared practical insights on C‑end link reliability, Alibaba’s system stability operations, Tencent Game’s cloud‑native SRE practices, and Ant Group’s chaos engineering, concluding with a Q&A and resource distribution.
A detailed investigation of an Elasticsearch timeout incident reveals how an overly aggressive node_concurrent_recoveries setting caused CPU saturation, disk I/O spikes, and shard relocation overload, and outlines the steps taken to isolate the faulty node and restore cluster performance.
This article explains the motivation, design, and implementation of JD’s lightweight Go‑based “localhost” tool that creates secure tunnels for remote debugging of locally running services, detailing its usage, core architecture, routing logic, and future enhancements.
This guide walks through setting up Azure OpenAI (GPT‑4) as a connector for Elastic Observability’s AI Assistant, covering prerequisites, Azure resource creation, connector configuration, URL formatting, and practical examples of log analysis and chat‑based troubleshooting.
An in‑depth review of Didi’s 12‑hour P0 outage reveals how a mistaken Kubernetes version downgrade during an in‑place upgrade caused master node failure, discusses cluster isolation, upgrade strategies, and the role of cost‑cutting pressures, offering practical lessons for large‑scale operations.
The article reviews the October 23 Yuque service outage, analyzes root causes such as a buggy upgrade tool and outdated storage, extracts operational lessons on testing, disaster recovery, high‑availability, communication, and advocates the KISS principle to simplify complex systems for improved reliability.
The article examines recent large‑scale service outages to illustrate how system complexity can be analyzed through four dimensions—component, structural, functional, and descriptive complexity—highlighting the need for comprehensive assessment to improve resilience and sustainable development.
This article introduces ten indispensable tools for operations engineers, detailing each tool's functionality, suitable scenarios, advantages, and real‑world examples, and includes practical code snippets to help automate, monitor, and manage infrastructure efficiently.
This article presents a comprehensive collection of R&D efficiency metrics and their calculation formulas, covering code integration, quality, productivity, reliability, maintainability, and deployment aspects, to help teams evaluate and improve development performance and operational effectiveness.
The article reviews Didi's late‑November 2023 service disruption, detailing the timeline of failures, official apologies, and expert analyses of six possible technical causes—including software bugs, server issues, third‑party failures, DDoS, other attacks, and ransomware—while highlighting the role of a Kubernetes upgrade and cost‑cutting pressures.
This article details the design, challenges, and monitoring strategies of Alibaba's travel billing system, explaining how a modular backend architecture, multi‑way reconciliation, full‑link monitoring, and a configurable expression engine enable near‑perfect bill accuracy and automated settlement for enterprise customers.
This article outlines the essential steps for embracing DevOps culture, emphasizing tailored strategies, deep understanding of CI/CD, clear role assignments, extensive automation, key performance metrics, and the critical role of quality assurance to achieve faster, reliable software delivery.
The article explains the common P0‑to‑PX incident severity hierarchy used in software development, detailing what constitutes a P0 crash versus lower‑level issues, notes that definitions can vary across organizations, and adds a personal perspective on Didi’s service reliability.
For the 13th League of Legends World Championship live stream on Bilibili, a comprehensive technical‑assurance framework—covering pre‑event traffic buildup, in‑event experience, and post‑event replay—mapped over 60 business functions, applied a traffic‑estimation model, executed fault‑injection drills, load tests, strict SOPs and change control, and real‑time monitoring, enabling 120 million viewers and a peak of 460 million concurrent users.
This article reviews how nineteen top Chinese companies applied the CAICT‑led DevOps Capability Maturity Model, detailing their assessment results, project improvements, and concrete performance gains such as higher release frequency, full test coverage, and streamlined operations across diverse industry sectors.
Amid a nationwide digital transformation push, seven leading Chinese joint‑stock banks have leveraged the CAICT‑led DevOps Capability Maturity Model to evaluate dozens of projects, revealing how DevOps improves IT efficiency, integrates resources, and supports business systems across the financial sector.
In the context of digital transformation, six leading Chinese telecom operators used the CAICT‑led DevOps Capability Maturity Model to evaluate 33 projects, improving IT performance, integrating resources, and supporting business systems, offering valuable industry‑wide best‑practice insights.
This article reviews how five leading Chinese insurance firms evaluated eight projects using the CAICT DevOps Capability Maturity Model, highlighting each company's implementation details, performance improvements, and the broader significance of the model for digital transformation and operational excellence in the insurance sector.
The article details how Beijing Mobile’s SRE Smart Operations team applied SRE principles, automation, and cloud‑native tools to transform traditional DevOps into a reliable, scalable operation, highlighting their fault‑prevention, monitoring, incident response, and continuous improvement practices that earned them the 2023 IT Technology Leadership award.
The article outlines common logging problems and presents a comprehensive six‑step strategy—including clear logging standards, systematic standard management, a unified SDK, centralized log management systems, regular standard reviews, and lifecycle deprecation—to transform chaotic logs into a reliable tool that boosts development efficiency.
The article examines Netflix’s platform engineering approach, detailing its centralized team structure, hub‑and‑spoke model, internal customer‑support system, productivity evaluation methods, challenges such as documentation, and ongoing efforts to improve developer experience and platform adoption.
This article explains why monitoring alert noise harms efficiency, presents metrics such as recall and accuracy, details rule‑based, blacklist/whitelist, ratio‑based, and intelligent noise‑reduction techniques, shares Java code examples, and shows measurable results after applying the governance process.
This article details Qunar's comprehensive overhaul of its monitoring platform—introducing second‑level metrics, redesigning storage with VictoriaMetrics, optimizing client and server data collection, and building a root‑cause analysis tool—to dramatically reduce order‑related fault discovery time from minutes to under one minute.
This article presents 36 concise data analysis models spanning six key business dimensions—Internet operations, strategy and organization, quality and production, marketing services, financial management, and human resources—to help analysts choose the right method for structured, logical, and effective insights.
This article explains how to reduce the impact of failures in distributed systems by simplifying service links, applying circuit‑breaker mechanisms, implementing graceful degradation, performing core‑link isolation, and, as a last resort, switching to a minimal MVP version to keep essential functionality alive.
This article defines digital transformation, introduces the Digital Transformation Framework (DTF) and its economic, risk and financial dimensions, and explains how asset‑management firms can redesign front, middle and back‑office functions, adopt composable enterprise models, and align culture, automation and API‑driven strategies to achieve sustainable, disruptive change.
This article presents a systematic approach for technology leaders to reduce and control cloud‑related expenses through cost‑optimization, detailed analysis, team organization, project planning, communication, and long‑term fine‑grained operational practices.
This article details the August 2023 Bilibili CDN failure, analyzes its root causes, describes the 1‑5‑10 emergency recovery framework, and presents cloud‑side SLB/BFS optimizations and edge‑side scheduling and fallback strategies that together restored service and improved future resilience.
This article recounts the author's experience preparing JD's Customer Data Platform (CDP) for the Double 11 shopping festival, detailing the platform's capabilities, business scenarios, capacity planning, stability and performance challenges, disaster‑recovery measures, and personal reflections on the intensive technical effort involved.
A cloud engineer mistakenly set a disk snapshot to public, exposing a major client’s data, rushed a rollback, and then reflected on the root causes, highlighting the need for strict review, visual tools, and risk‑aware practices in high‑risk operations.
This article examines how six major Chinese state-owned banks leveraged the CAICT‑led DevOps Capability Maturity Model to assess dozens of projects, improve IT efficiency, integrate resources, and achieve measurable gains in delivery speed, quality, and security across agile development, continuous delivery, operations, and risk management.
This article analyzes the rising compute demand in modern deep‑learning‑driven search systems, proposes a micro‑ and macro‑level adaptive power‑allocation framework, models the optimization problem with cost, time, and feasibility constraints, and details an elastic cascading architecture that dynamically balances resource usage, system state, and traffic value to achieve higher ROI and stability.
This article explains what SaaS is, outlines its business value, and walks through the full technical stack—from entry points and access layer to business, data, and operations layers—while comparing shared, dedicated, and hybrid deployment models and future challenges.
This article explains the concept and importance of MTTR (Mean Time To Repair), shows how to calculate it, and provides a comprehensive set of monitoring, alerting, rapid mitigation, tool‑assisted analysis, and team coordination techniques to significantly shorten incident resolution time and improve system reliability.
Everbright Securities’ Sunshine E‑Office project passed the CAICT DevOps Continuous Delivery Level‑3 assessment, showcasing how standardized DevOps practices and tool empowerment can dramatically improve development efficiency, quality, and security, while driving digital transformation across the financial sector.
At the 2023 GOPS Global Operations Conference in Shanghai, China Information Communication Research Institute announced that CICC Wealth's unified access and authentication project passed the DevOps Technical Operations Level‑2 assessment, showcasing how standardized DevOps practices and tool empowerment can dramatically improve quality, efficiency, and market competitiveness in the financial sector.
This article explains Site Reliability Engineering (SRE) as a collaborative methodology, outlines its stability goals measured by MTBF and MTTR, details how SLI/SLO and the VALET selection guide fault detection, and shows how error budgets quantify reliability work and drive precise alerting.
This article compares the open‑source monitoring solutions Prometheus and Zabbix, outlining their histories, architectures, data collection methods, scalability, storage models, configuration complexity, community activity, and suitability for different environments such as traditional servers versus cloud‑native container platforms.
The article outlines how game QA and third‑party providers can improve cooperation by aligning basic performance concepts such as TPS, QPS and concurrency, selecting appropriate rate‑limiting strategies, establishing precise monitoring and alerting, and preparing clear incident‑response and delivery standards.
The 21st GOPS Global Operations Conference in Shanghai unveiled the latest DevOps capability maturity assessment results from major Chinese banks, highlighting pioneering evaluations, detailed improvements across configuration, monitoring, and user experience, and introducing the comprehensive DevOps maturity model that guides digital transformation.
The Industrial and Commercial Bank of China software development center created an SRE panoramic monitoring view system that unifies data channels, standardizes metrics, offers multi‑dimensional dashboards, and introduces an intelligent Ops Assistant, dramatically improving fault detection, response speed, and cross‑team operational efficiency.
The article describes how Ant Group’s consumer credit technology platform leverages AI and cloud‑native architectures to achieve ultra‑fast operations, precise fund verification, large‑scale simulation, and seamless migration for dozens of financial institutions, addressing the massive technical challenges of internet‑scale credit services.
This article details Guangda Securities' successful DevOps assessment at the 2023 GOPS Global Operations Conference, describing how adopting standardized DevOps practices and a continuous delivery pipeline elevated their development efficiency, quality, and competitiveness in the financial technology sector.
At the 2023 GOPS Global Operations Conference, China Merchants Securities showcased its successful DevOps assessments—earning multiple Level 3 ratings in agile development and continuous delivery—demonstrating how standardized processes, tool empowerment, and a unified digital management system can accelerate digital transformation and boost market competitiveness.
This article explains how Linux kernel parameters, memory size, and file descriptor limits determine the maximum number of TCP connections a server or client can handle, provides configuration examples for increasing those limits, and discusses practical considerations such as port ranges and connection overhead.
This article explains how to design and deploy a complete monitoring solution using Prometheus, various exporters, Grafana, and Alertmanager on Docker and Kubernetes, covering installation, configuration, visualization, and best‑practice tips for reliable operations monitoring.
In a detailed Q&A, Zhengzhou Yisheng Information Technology shares how its two exchange‑focused platforms achieved the DevOps Continuous Delivery Level‑3 assessment, highlighting process improvements, metric gains, architectural choices, challenges overcome, and future plans for broader digital transformation.
The article explains platform engineering as an evolution of DevOps that emphasizes internal developer platforms, clear organizational goals, reduced friction for engineers, and practical, incremental solutions rather than over‑reliance on complex tools, highlighting its rising popularity and distinct approach.
The 21st GOPS Global Operations Conference in Shanghai unveiled the latest DevOps capability maturity assessment results, highlighting first‑time evaluations across exchanges, securities and fund companies, detailed improvements in delivery speed, test coverage and automation, and introducing the comprehensive DevOps standards now adopted industry‑wide.
The 21st GOPS Global Operations Conference in Shanghai unveiled the latest DevOps capability maturity assessment results, highlighting how major Chinese banks and financial institutions have adopted DevOps standards to improve technology operations, agile development, security, and system tooling across multiple projects.
This guide presents essential kubectl commands for creating and managing Deployments, Pods, Services, and Ingresses, checking cluster and node status, running container applications, updating images and resource limits, and handling log collection and monitoring in a Kubernetes environment.
The October 23 Yuque outage, caused by a buggy upgrade tool and outdated storage hardware, highlighted the importance of thorough testing, robust disaster‑recovery, high‑availability architecture, clear communication, continuous learning, and applying the KISS principle to simplify complex systems and improve operational stability.
Dongguan Securities’ ‘Zhangzhengbao Service Platform’ passed the China Information and Communication Research Institute’s DevOps Capability Maturity Model Level‑3 Continuous Delivery assessment, showcasing how standardized DevOps practices, automated pipelines, and micro‑service architecture dramatically improved development efficiency, code quality, and operational agility, positioning the firm as a domestic leader in financial technology.
This article outlines practical steps, challenges, and results of implementing data table compliance governance in a fast‑growing data warehouse, covering standards redefinition, decommissioning unused tables, metric reuse, ODS penetration reduction, and ongoing maintenance strategies.
Zhejiang Rural Commercial Bank’s two flagship projects passed the Level‑3 Continuous Delivery assessment, showcasing how standardized DevOps practices, tool empowerment, and agile transformation can dramatically improve software quality, delivery speed, and overall competitiveness in the banking sector.
This article details Zhengzhou Commodity Exchange’s technology company's successful DevOps continuous‑delivery Level 3 assessment, sharing interview insights, performance metrics, implementation challenges, and future plans that illustrate how standardized, automated DevOps practices boost delivery speed, quality, and digital transformation.
The article outlines eBay’s key scalability best practices—including functional partitioning, horizontal sharding, eliminating distributed transactions, aggressive asynchronous decoupling, intelligent caching, and pervasive virtualization—to illustrate how large‑scale web systems can achieve linear or sub‑linear growth while maintaining availability and performance.
This guide explains Loki's core concepts, deployment steps for Promtail and Loki, Grafana integration, label‑based indexing, handling dynamic and high‑cardinality tags, and query optimization techniques, providing a complete roadmap for building a cost‑effective, scalable log aggregation system.
This article examines the rapid growth of the Chinese IT operations market, explains why AIOps faces trust challenges due to opaque deep‑learning models, and presents AsiaInfo's transparent‑model and post‑hoc explanation engine together with three concrete explainable root‑cause analysis methods, concluding with future outlooks for trustworthy AIOps.
This guide explains the concepts of ZooKeeper cluster roles and provides step‑by‑step instructions, including configuration file edits and command‑line operations, to set up a pseudo‑cluster on a single machine for testing and learning purposes.
This article walks through practical methods for troubleshooting slow service alerts, diagnosing Redis performance bottlenecks, and reproducing issues with local demos and load simulations, offering concrete metrics, command‑line checks, and mitigation strategies such as scaling, rate‑limiting, and pipeline optimization.
This article explains Memcached’s architecture, key characteristics, suitable and unsuitable scenarios, memory management and LRU mechanisms, version details, and provides a comprehensive guide to monitoring its performance and health using Prometheus and Alibaba Cloud ARMS dashboards.
This guide explains why monitoring systems need self‑monitoring, how to leverage their own /metrics endpoints for internal health checks, and how to supplement them with a lightweight external monitor using catpaw plugins and FlashDuty for robust alerting.
Learn how to diagnose and resolve frequent Kubernetes pod status issues such as ContainerCreating, ErrImagePull, Pending, CrashLoopBackOff, and UnexpectedAdmissionError by examining Docker services, storage mounts, ConfigMaps, image repositories, and node resources, with practical examples and command‑line solutions.
This step‑by‑step guide shows how to install FastDFS on a Linux server, configure its tracker and storage nodes, integrate it with Nginx, and expose the file service to the Internet using the cpolar tunneling tool, all with detailed commands and configuration files.
This article explains how systematic test case creation, clear verification steps, and disciplined bug tracking improve testing efficiency, ensure comprehensive coverage of core system logic and data flows, and help teams identify root causes to continuously raise software quality despite limited resources.
This article breaks down high‑availability system design into six critical layers—architecture, development standards, application services, storage, product safeguards, and operations—offering concrete practices such as capacity planning, fault‑tolerant patterns, monitoring, and incident‑response strategies to achieve four‑nine (99.99%) uptime.
The 2023 China International Service Trade Fair’s Enterprise Digital Transformation Forum highlighted ICBC Fund’s award‑winning DevOps‑based trading management platform, showcasing how a high‑cohesion, low‑coupling architecture and dual mid‑platform foundation enable online operations, risk compliance, and industry‑first DevOps certification.
This article describes how Qunar implemented an automated root‑cause analysis system for flight‑ticket transaction interception, detailing the problem background, system research, a custom algorithm focusing on explanatory power, performance optimizations that reduced analysis time from five minutes to under ten seconds, and the resulting operational improvements.
The article explains how Home Depot (THD) adopted the VALET pattern language—Volume, Availability, Latency, Error, and Ticket—to unify service‑level objectives, automate data collection, build dashboards, and improve SRE practices across its massive retail and e‑commerce infrastructure.
This article introduces Ansible as an open‑source automation tool, explains its core architecture and workflow, and provides detailed usage of its seven primary commands with examples, helping readers quickly grasp how to configure, deploy, and manage systems efficiently.
This article explains the essential fault metrics MTTR, MTBF, and MTTF, their definitions, calculations, and practical importance for SRE and operations teams to improve system availability, guide maintenance strategies, and make data‑driven reliability decisions.
This article provides a comprehensive overview of Apache Kafka, covering its architecture, core concepts such as topics, partitions, and replicas, common operational commands, and practical performance‑tuning tips for high‑throughput, low‑latency streaming workloads.
This article reviews eight popular Docker monitoring solutions, detailing their key features such as performance metrics, dashboards, alerting, capacity planning, log analysis, and ease of setup, helping you choose the right tool for container observability.
The article outlines how organizations over a hundred engineers must go beyond merely copying DevOps practices by adopting three progressive steps—normalization, standardization, and platformization—to achieve measurable, scalable efficiency, and concludes with a promotional notice for a Python‑based continuous deployment training course.
This article explains how intelligent supply‑chain concepts can be applied to the pharmaceutical sector by outlining the industry's supply‑chain structure, digital transformation opportunities and challenges, data‑capability monetization models, and the use of AI and knowledge‑graph technologies to capture growth opportunities.
This article describes a real‑world OceanBase failure scenario where a core dump fills the data disk, and provides a step‑by‑step guide to shrink datafiles to 80% of disk capacity, adjust parameters, reduce tenant replicas, offline and reinstall OBServer instances, and finally restart the cluster.
This article explores the four pillars of technical operations—Dao (philosophy), Fa (methodology), Shu (techniques), and Qi (tools)—detailing concepts from business continuity to SRE, FinOps, GitOps, CMDB design, and the underlying logic of modern ops platforms.
The document proposes a declarative, multi‑OS baseline management platform that groups servers, supports gray‑release rollouts, monitors state, and automatically restores configurations, extending open‑source tools to provide versioned, conditional, and auditable system‑environment control across a large‑scale infrastructure.
This article outlines the most common production failures—including network, server, database, software bugs, security vulnerabilities, storage, configuration errors, and third‑party service issues—and provides detailed steps for detection, investigation, and resolution to ensure system stability and reliability.
This article shares Huolala's experience designing a unified monitoring platform UI, covering the evolution from open‑source dashboards to a fully self‑developed solution, simplification of PromQL, computed metrics, log and trace integration, and the challenges of alert configuration and visualization.
The Network Quality Monitoring Center is a large‑scale system that deploys lightweight agents on every server to issue coordinated ICMP ping probes, a controller to generate and distribute topology‑aware PingLists, and a storage‑analysis module that aggregates latency and loss data for real‑time visualization, alerting and troubleshooting, while addressing load‑balance, ingestion concurrency, and future extensions such as UDP/TCP probes.
The article explores why building system stability is crucial by linking entropy theory to software reliability, introduces the availability formula, discusses common pitfalls and industry practices, and proposes a three‑stage governance framework—prevention, mitigation, and post‑mortem—to systematically improve operational resilience.
During system development and operations, missing exception stack traces can hinder troubleshooting; this article explains how JIT compiler optimizations like Fast Throw cause stack traces to disappear, outlines conditions for Fast Throw, shows performance impact, and demonstrates how to locate root causes using logs and metrics.
During a night‑time outage, a XCache (Codis + Pika) instance hung due to massive write load triggering low‑level protection, causing Sentinel to switch masters; the proxy’s accept queue filled with timed‑out sockets, blocking new connections, so scaling the proxy layer and expanding capacity restored service while prompting automation, health‑check, and queue‑overflow alerts.
This guide explains how Linux stores user and group information in /etc/passwd, /etc/shadow, /etc/group, and /etc/gshadow, and provides detailed usage of commands such as useradd, usermod, userdel, groupadd, groupmod, and gpasswd for creating, modifying, locking, and deleting accounts and groups.
Learn how to efficiently monitor multiple Kubernetes pods by installing and using two lightweight, real‑time log aggregation tools—Kubetail and Stern—including installation steps for Homebrew, Linux, and Zsh, command‑line options, color output, and practical usage examples.
This guide explains how to export Docker images for MySQL and Zabbix components, transfer and load them on an offline Kirin production server, and then run MySQL, Zabbix Java gateway, Zabbix server, and Zabbix web containers with the necessary configuration and linking.
This article shares practical Linux and Vsphere techniques—including using rsync for rapid bulk deletions, detecting new iSCSI disks without reboot, safeguarding rm commands, mounting remote filesystems with SSHFS, and adding users to supplementary groups via gpasswd—to streamline daily operations.
Kubernetes is moving its Debian and RPM package repositories to the community‑owned pkgs.k8s.io domain, deprecating the old apt.kubernetes.io and yum.kubernetes.io sources, and the article explains the timeline, impact on end users and distributors, and step‑by‑step migration guidance.
This article introduces the open‑source web platform Mayfly-go, explains its key features for Linux system, MySQL, PostgreSQL, Redis, and MongoDB management, provides step‑by‑step installation and configuration instructions, and demonstrates how to use its project, machine, database, and system administration capabilities.
