This article examines how a large‑scale e‑commerce transaction platform achieved high availability through iterative architectural evolution—from early .NET monoliths to vertical and horizontal micro‑service splits—highlighting practical strategies for fault detection, rapid recovery, scaling, and operational best‑practices.
This article explains why Docker services need monitoring, distinguishes black‑box and white‑box approaches, compares self‑hosted and SaaS monitoring stacks, and reviews key components and popular tools such as Prometheus, InfluxDB, Grafana, Datadog, and Sysdig.
A comprehensive overview of common Linux daemon processes, detailing each service’s purpose, typical use cases, and key configuration notes for system administrators seeking to understand and manage background services effectively.
This article shares the author’s experience building fault‑tolerance for Tencent’s activity operations platform, covering retry strategies, automatic removal of unhealthy machines, timeout tuning, asynchronous processing, anti‑replay mechanisms, service degradation, service decoupling, and business‑level safeguards to reduce manual alarm handling and improve system robustness.
This article, based on the Efficient Operations Community Talk, outlines the evolution of e‑commerce systems, the challenges faced during rapid growth, and how LeTV’s e‑commerce cloud leverages micro‑service architecture, container technology, and hybrid cloud solutions to address scalability, security, and operational efficiency.
This article explains what deployment specifications are, outlines the key components of a good spec, shares a real-world CodeDeploy example, and provides practical steps for designing, building, and rolling out deployment standards that balance flexibility, non‑intrusiveness, and ease of use.
This article shares Tencent’s experience building fault‑tolerant mechanisms for the AMS activity platform, covering retry strategies, automatic machine exclusion, timeout tuning, service isolation, asynchronous processing, anti‑replay safeguards, and operational best practices that transformed a million‑request service into an 800‑million‑request system.
This guide explains how to use the lightweight ngxtop tool to monitor Nginx in real time, showing frequent requests, high‑traffic IPs, and providing useful command examples for filtering logs, ordering results, and installing the utility via pip.
At the GOPS2016 conference, speaker Cui Xiaochun likens the invention of the horse stirrup to modern automation, tracing the evolution of operations from manual scripts to AI-driven intelligent systems, and argues that embracing AI is the next revolutionary step for ops teams.
A 2005 fat‑finger error by a Mizuho trader triggered a ¥400 billion loss, leading to a landmark court case that clarified liability for software bugs in financial systems and highlighted the need for rigorous testing and evidence preservation.
This article explains how to design and implement a unified operations monitoring and alerting platform by combining Ganglia for data collection with Centreon for alerting, covering architecture layers, module functions, integration steps, and practical Q&A for large‑scale deployments.
During an exclusive visit to Facebook’s Menlo Park campus, the author uncovers the company’s sophisticated release engineering practices—including the HipHop optimizer, a custom BitTorrent‑based deployment system, continuous testing, and a unique “Hotfix Bar” culture—revealing how billions of daily requests are reliably delivered at massive scale.
This article outlines a comprehensive design for a high‑performance, unified operations monitoring platform, detailing a six‑layer architecture, the roles of data collection (using Ganglia), data extraction, and alerting modules (with Centreon), and provides practical integration tips, deployment diagrams, and Q&A for large‑scale environments.
This article explains a systematic, waiting‑point‑based approach to enterprise Java performance tuning, covering load‑test design, analysis of existing versus new applications, hierarchical and technical waiting points, pool and cache sizing, and a back‑tuning workflow to achieve measurable improvements.
This article explains Vipshop’s multi‑layer monitoring architecture, detailing system‑level metrics, application‑level tracing with the Mercury platform, and business‑level KPI dashboards, while describing the data pipelines that collect, process, and alert on distributed logs to ensure reliable operations.
This article explains how to build an operations evaluation model within a quality competitiveness framework, detailing a three‑step process for defining metrics, evaluation methods, and quantification, and highlighting essential evaluation points, attention areas, and data collection practices for product operations.
The article examines various unconventional data‑center cooling methods—including lake‑water, seawater, waste‑water, mineral‑oil, and hot‑water systems—highlighting their impact on Power Usage Effectiveness (PUE) and overall energy efficiency across major tech companies.
The article describes a three-dimensional, automated, visual monitoring approach for high-availability systems, detailing a five-layer monitoring model, automated log collection using Logstash-Redis-Elasticsearch, and visualization techniques that together reduce fault-locating time and improve operational efficiency.
This article provides a detailed overview of Stack Overflow's 2015 architecture, covering daily traffic growth, hardware upgrades, redundancy principles, DNS and ISP routing, HAProxy load balancing, IIS/ASP.NET web layer, Redis caching, WebSocket services, Elasticsearch search, SQL Server databases, and the open‑source tools that support the platform.
The article explains how functional separation and degradation techniques—distinguishing core from non‑core services, isolating them logically and physically, and implementing manual or automatic fallback mechanisms—help maintain high availability in distributed systems during traffic spikes or component failures.
This talk explores how Meipai designed a scalable, reliable database architecture that balances SQL and NoSQL technologies to support rapid product iteration and high‑peak traffic, highlighting key considerations for storage service reliability and performance optimization.
This article outlines a complete performance‑testing workflow—including requirement analysis, overall and detailed design, environment selection, execution, result collection, and bottleneck analysis—using a wallet "red‑packet" project that must handle tens of thousands of requests per second.
This article examines the common pitfalls faced by engineers in fast‑growing startups—from poor project planning and rushed code refactoring to unclear product requirements, weak organizational processes, hasty technology choices, operations overload, and people‑related challenges—offering practical guidance to navigate each issue.
The article details Stack Overflow’s 2016 infrastructure upgrades—including hardware, networking, load balancing, caching, database, and service layers—that enabled the site to process over 61 million daily requests while reducing processing time by hundreds of hours.
This guide explains how to install, configure, and use the sar (System Activity Reporter) tool on Linux to monitor CPU, memory, I/O, and load metrics, helping you pinpoint performance bottlenecks through various command options and detailed reports.
This article presents a comprehensive overview of Dianping's operations architecture, detailing team organization, multi‑data‑center infrastructure, monitoring layers, automation tools, configuration management systems, incident analysis, lessons learned, and future directions such as Docker and PaaS adoption.
This article explores the DRY principle in software development, explains why many open‑source projects violate it, and provides practical guidance on selecting, using, and customizing open‑source solutions through real‑world case studies, focusing on business fit, maturity, operational capability, and safe integration.
This article provides Java developers with a concise reference of essential Linux shell commands, covering process inspection, file manipulation, permission changes, compression, networking checks, remote access, and other common operations needed for interacting with Linux servers during development and deployment.
The article explains why developers need to understand operations—covering resource usage, fault handling, platform basics, and essential ops tools—so they can write maintainable code, avoid common pitfalls, and collaborate effectively with ops teams for reliable, high‑performance services.
This article provides a comprehensive overview of DevOps, explaining its purpose, cultural challenges, core principles such as automation, standardization, and configuration, its relationship with cloud, lean and agile, practical steps, metrics, and how it transforms IT delivery into an end‑to‑end business value pipeline.
The article critiques the hype around operations automation, arguing that many tasks can be handled with simple shell scripts, that automation should solve error‑prone manual work rather than replace thoughtful architecture, and that choosing the most convenient tool is more valuable than chasing trendy solutions.
Drawing on fourteen years of hands‑on experience, the article chronicles how a website’s architecture matures from a simple personal homepage to a billion‑page‑view enterprise system, highlighting the essential principles, design patterns, operational practices, and scalability strategies that underpin successful large‑scale web platforms.
Meituan’s internal RPC services, largely built on Thrift, required a streamlined pressure‑testing solution, leading to the development of a custom tool that automates traffic capture, provides an intuitive UI, aggregates metrics via InfluxDB, and supports both Thrift and HTTP workloads, addressing the shortcomings of existing open‑source options.
This article provides a comprehensive overview of Hyper-V, covering its extensive operating system support, virtual networking, management integration with System Center, dynamic memory, storage options, VM conversion tools, and key SMB 3.0 features for high‑availability and performance in virtualized environments.
The article outlines VIP's transition from a simple outlet‑style e‑commerce platform to a multi‑brand flash‑sale service, detailing each architectural phase—from a monolithic LAMP stack through vertical silo and distributed service‑oriented designs—to a cloud‑native, platform‑plus‑application model that supports scalable, high‑availability operations.
This article introduces dstat, a versatile Linux monitoring tool that consolidates vmstat, iostat, and ifstat functions, demonstrates its default and customizable outputs, shows how to identify top resource‑consuming processes, and provides simple installation steps for CentOS.
This article explores a new platform capability model that integrates business support, development support, and operational control, arguing that a platform should be treated like a product with clear responsibilities, measurable outcomes, and a closed‑loop approach to sustain long‑term value.
As cloud computing reshapes IT, traditional operations roles face unprecedented disruption, but by embracing cloud‑focused responsibilities, niche industry needs, or even a complete career pivot, ops professionals can secure their future within the next five to ten years.
This article walks through a real‑world Redis timeout problem, examining client‑side socket settings, server configuration, system metrics, and kernel TCP parameters to pinpoint why connections are dropped and how adjusting timeout and tcp‑keepalive resolves the latency.
This article examines recent high‑profile internet outages caused by human error, explores why operations teams are especially prone to mistakes despite automation and standards, and offers practical strategies—such as hiring the right people, fostering safety awareness, and turning professionalism into habit—to reduce future incidents.
This article explores how a people‑centric mindset can transform operations by defining a three‑layer framework, clarifying why human factors matter, and offering concrete process, technology, and organizational practices such as streamlined approval flows, voice‑alert systems, and Docker‑based continuous deployment.
This article examines why many operations engineers feel unhappy, identifies four personal‑management problems—over‑pursuing tech, mis‑prioritizing tasks, poor communication, and chronic complaining—and offers concrete, actionable suggestions to improve productivity, satisfaction, and team collaboration.
This article explores why efficient operations are hard to achieve, identifies common pitfalls such as unclear responsibilities, communication gaps, and resource mismatches, and presents a practical framework—including clear roles, professional processes, and a good service interface—to help operations teams become more effective and satisfied.
The article argues that traditional IT operations are facing a crisis and proposes Operations 2.0—a service‑oriented, business‑aware model that leverages cloud, open‑source and automation to shift focus from technical output to reliable, value‑adding services, outlining why it is essential and how to implement it.
This article explores the benefits and trade‑offs of performance optimization, outlines single‑application and structural optimization approaches, details bottleneck identification methods, common tuning techniques, and illustrates architectural evolution with diagrams to guide effective ops improvements.
This article shares practical insights on achieving high availability by understanding goals, decomposing requirements, designing resilient architectures, ensuring operability, testing rigorously, and reducing release risk, illustrated through the multi‑stage evolution of a transaction system.
The article examines how large‑scale web sites handle massive image resources, tracing the progression from simple single‑machine storage to clustered virtual directories, shared UNC storage, and finally a FastDFS‑based distributed file system combined with CDN acceleration, highlighting the architectural trade‑offs and operational considerations.
While Jenkins offers powerful deployment plugins, this article explains why a standalone deployment system remains essential for continuous delivery, covering decoupling builds, managing complex environments, supporting varied deployment strategies, enforcing standards, gathering operational data, and enabling service-oriented deployment across teams.
The article documents a LAN test copying a single 200 GB file with scp, rsync, wget, and bbcp, compares their speeds, provides installation steps for bbcp, explains key options, and concludes which tool works best for large file transfers.
This article explains how Docker’s build‑ship‑run model bridges front‑end development and containerization, covering Docker’s history, core concepts, a sample Dockerfile for a Node.js app, and practical scenarios where Docker improves environment consistency, resource efficiency, and scalability.
When a massive container explosion threatened Tencent's Tianjin data center, the operations team executed a 24‑hour, continent‑wide user migration that moved over 200 million QQ users to Shenzhen and Shanghai without service interruption, showcasing unprecedented disaster‑recovery capabilities.
Immutable Infrastructure treats every server or container as a read‑only unit that is replaced rather than modified, offering repeatable configuration, faster CI/CD, easier rollback, and reduced operational complexity, while requiring stateless applications and automated provisioning templates to succeed.
This article chronicles the two‑month development of WeChat’s backend from its inception, detailing the design of its message model, data‑sync protocol, three‑tier architecture, asynchronous queues, rapid scaling, platformization, multi‑data‑center deployment, disaster‑recovery strategies, performance optimizations, security hardening, and emerging resource‑scheduling challenges.
The article examines the trade‑offs between incremental and full deployment, outlining their workflows, advantages, and challenges, and concludes that full deployment is generally preferable for stateless units while incremental methods remain useful for stateful components like databases.
This article explains China's green data‑center policies, the importance of PUE, and demonstrates through calculations and real‑world Dalian case studies how natural cooling can halve cooling energy use, lower PUE from 2.5 to 2.0, and save millions in electricity bills.
This article outlines ten practical guidelines—ranging from test environment consistency and dedicated hardware to network capacity checks, OS tuning, realistic workloads, proper test duration, and comprehensive result reporting—to ensure reliable and reproducible HTTP server performance benchmarks.
This guide explains how to automatically rotate Nginx logs each day by renaming the current log file at midnight, creating dated backup directories, and signaling the Nginx master process with USR1 using a simple Bash script scheduled via a cron job.
The article explains why service degradation is essential for large‑scale shopping events, outlines its different dimensions such as page, business module, and remote service downgrade, and describes both manual and automatic implementation methods to maintain system availability under heavy load.
This article explores how logs serve as vital diagnostic tools in operations, detailing their definition, practical uses for performance and security monitoring, step‑by‑step analysis techniques, and recommendations for free and commercial log analysis solutions.
A seasoned operations leader discusses how e‑commerce operational support differs from general internet applications, covering longer support chains, consistency models, seasonal traffic spikes, team role separation, mobile‑internet challenges, future planning, and the rise of enterprise‑level ops services.
Learn how to use a single netstat pipeline command to count established connections on port 80, giving you an immediate view of your web server's concurrent connection load for better performance monitoring and troubleshooting.
This article explains the origins, core features, design principles, and deployment resources of Jumpserver, an open‑source Python‑based bastion host that simplifies batch account management, command auditing, and web‑based terminal access for operation engineers.
Learn three powerful shell commands that reveal the most active IPs, the most requested URLs, and the largest files in an Apache access_log, helping you quickly detect anomalies, optimize resources, and improve site performance.
This article explains how JD.com’s Order Fulfillment Center (OFC) was built, re‑engineered, and continuously optimized to handle massive order volumes during major sales events, covering its architecture, migration from .Net to Java, distributed task queues, flow control, and operational practices that ensure reliability and scalability.
This article shares Tencent’s QQ operations team insights on handling legacy issues, standardizing package and configuration management, leveraging the ZhiYun automation platform, and applying capacity management and fault‑root analysis techniques to boost efficiency and reduce costs.
In this interview, seasoned ops veteran Zhijin discusses the similarities and differences between traditional and internet operations, the challenges of building private clouds in finance, advice for ops professionals considering entrepreneurship or job changes, and predicts a future where private, public, and industry clouds coexist.
This guide details how to set up a Cisco 3560G three‑layer switch to support two groups with separate internet exits by defining VLANs, assigning IP addresses, creating ACLs, configuring route‑maps, and applying policy routing to direct traffic through the appropriate upstream routers.
This article summarizes a seasoned operations expert’s insights on data‑center management, covering the evolution from ad‑hoc automation to a closed‑loop CMDB‑driven system, the six key capabilities for future data centers, and practical definitions of operations, automation, and DevOps.
This article explains how a unified LAMP-based solution enables fast development, automated testing, and streamlined operations for large‑scale online services, detailing the architecture’s layers, tooling, and future directions for standardization and platformization.
This guide explains how email systems work—from DNS MX record lookup and server roles like MUA, MTA, MDA, and MRA to the detailed steps of sending, receiving, authentication, encryption, and spam filtering—providing operations engineers with a comprehensive understanding of mail infrastructure.
The article critiques the growing DevOps and full‑stack developer hype, arguing that forcing engineers to juggle development, operations, QA, and DBA tasks devalues specialized work, creates unrealistic expectations, and ultimately harms both productivity and software quality.
The article argues that the growing DevOps culture and the demand for “full‑stack” developers force engineers, especially in startups, to juggle multiple specialized roles—development, QA, operations, DBA—leading to inefficiency, burnout, and a dilution of true software craftsmanship.
Nagios is an enterprise‑grade, open‑source monitoring framework that tracks server, service, and network metrics such as CPU usage, memory, disk space, and network throughput, alerts via email or SMS on anomalies, and consists of a core, plugins, and extensions, though it lacks built‑in reporting and has configuration limitations.
This article details Meituan's supply‑chain system evolution—from its early manual processes to a highly automated, multi‑business platform—highlighting architectural redesign, dynamic forms, and product‑center structures that enabled rapid scaling and diversified service support.
JD’s Order Fulfillment Center (OFC) evolved from a small data‑transfer team into a highly scalable, distributed architecture that handles massive order volumes during events like 618 and Double‑11, employing Java migration, service decomposition, flow control, and robust operations to ensure data consistency and rapid delivery.
A sudden high‑load alert on a web server was traced to Nginx’s massive write operations on fastcgi_temp files, revealed through top, iotop, lsof, strace and iostat, and temporarily resolved by redirecting fastcgi_temp to memory.
This article explains SSD fundamentals, including flash memory principles, device composition, controller functions, common pitfalls, performance degradation causes, and best practices for monitoring and maintaining SSDs in enterprise environments, ensuring reliability and data integrity.
The Linux ‘w’ command provides a richer view of system load than ‘uptime’, displaying average load over 1, 5, and 15 minutes, detailed information about logged‑in users, their CPU usage, current activity, and offers guidelines for interpreting load values on single‑ or multi‑CPU systems.
JD.com’s senior engineers detail how a combination of massive Docker‑based cloud migration, multi‑center transaction architecture, intensive 60‑second recovery drills, and AI‑driven personalization via the JD Brain enabled the platform to handle the unprecedented traffic and data demands of the 11.11 shopping festival.
This article explores how operations managers can define their positioning, adopt multiple leadership roles, and implement effective performance management practices—including clear goal setting, task allocation, coaching, and systematic review—to boost team efficiency and stability.
This guide explains how to install and use the lightweight Linux tool http_load to benchmark web server performance, covering preparation of URL lists, key command‑line parameters, example test runs, and interpretation of the detailed output statistics.
This article shares practical insights from a seasoned KVM specialist on how to monitor hardware, set up alerts, design disaster‑recovery strategies, choose optimal software and hardware, and evaluate public‑cloud providers when migrating workloads to a virtualized environment.
This article shares the author's personal journey of conceiving, planning, writing, and publishing a technical book on Puppet, detailing the motivations, step‑by‑step process, challenges, case studies, and the professional benefits gained from turning the writing effort into tangible value.
The article explains Meituan's supply chain (SCP) process, detailing its role in converting merchant agreements into electronic contracts, the complex data structures, flexible sales models, dynamic auditing, and how automation, workflow, and product‑center modeling address these challenges to dramatically reduce costs and improve efficiency.
This article shares practical production‑level KVM virtualization techniques, covering CPU binding and host‑passthrough, memory management, network optimization with Open vSwitch, storage choices, VM time drift handling, and resource limiting via CGroup, offering actionable insights for reliable, high‑performance virtualized environments.
An in‑depth guide walks readers through building a comprehensive monitoring system for small‑to‑medium enterprises, covering hardware, system, application, network, security, traffic analysis, business metrics, log aggregation, automation, visualization, and practical integration with tools like Zabbix, IPMI, ELK, and Smokeping.
This article examines the characteristics and billing models of multi‑data‑center networks, analyzes external traffic patterns, identifies challenges in optimizing Internet‑facing bandwidth, and proposes practical scheduling strategies to better utilize idle bandwidth and reduce carrier costs.
This article explores why operations teams must adopt a people‑centric mindset, outlines a three‑layer framework of framework, blood (processes & policies) and interface, and provides practical steps for improving processes, technology, and organization to boost efficiency and employee satisfaction.
JobCenter is a distributed task coordination platform that replaces crontab with a unified scheduling, monitoring, and alerting system, enabling e‑commerce teams to manage thousands of web‑service‑based jobs, ensure reliable execution, and gain clear visibility into task performance.
This article provides a comprehensive tutorial on installing the Clip Server (Apache, PHP, MySQL), configuring its virtual host, setting up the Clip SDK with Python, and using various Clip commands to manage IP relationships, all illustrated with command examples and screenshots.
This article explains how Nginx Plus’s commercial stream module enables TCP load balancing, detailing configuration steps, underlying routing algorithms, health monitoring, connection handling, and performance considerations, while comparing it to HTTP load balancing and other layer‑4 solutions.
The article critically examines how the DevOps and full‑stack trends, driven by startup culture, force developers to juggle multiple roles, leading to overwork, reduced focus, and higher costs, while also highlighting nine companies that successfully practice DevOps.
This article recounts a series of July Redis incidents—including network‑card saturation, connection‑limit exhaustion, suspected split‑brain, Bgsave‑induced OOM, and master‑restart data loss—detailing the migration to Redis Cluster with a Smart Proxy, the challenges faced, and actionable remediation strategies.
This article presents a comprehensive set of logging best‑practice rules—from defining log levels and classifications to using RequestIDs, monitoring alerts, and managing log size—aimed at improving system reliability, troubleshooting speed, and operational efficiency.
This article outlines ten practical management principles—from clarifying structure and goals to visualizing work and demanding results—that help leaders improve team accountability, transparency, and continuous improvement in operational environments.
Tencent’s operations team explains how its SET‑based NoSQL clusters deliver ultra‑low latency, high availability, and seamless disaster recovery for billions of users, detailing deployment models, synchronization mechanisms, cost‑saving techniques, and the Data‑as‑Service approach that underpins its massive social platforms.
The article chronicles Tencent's technological evolution across four stages—from massive user growth to flexible platforms—highlighting the strategic use of efficient architectures, dynamic operations, and cloud services that enabled it to support billions of users while maintaining performance and reliability.
In a candid Q&A, industry veterans discuss the fundamental differences between internet and traditional banking operations, share experiences transitioning between sectors, and outline strategies to eliminate difficult data‑center maintenance, highlighting risk‑focused versus growth‑driven approaches.
The article explores Randy Shoup’s insights on Google’s DevOps model, focusing on Dr. Spear’s four capabilities—especially the ways high‑efficiency companies rapidly detect issues, swarm to resolve them, disseminate new knowledge company‑wide, and adopt a development‑led leadership approach.
This article summarizes a DevOps capability model, defining engineering and architectural abilities, outlining OS and application system capability models, and offering practical advice on how Dev and Ops should evolve together to achieve reliable, reusable, and automated operations.
