Building Automated Ops with Modular “Lego” Models: Strategies, Tools, and Methodology

Main Participants

老王@UC

谭志宇@胡莱游戏

李才荣@百度

Editor

董伟@SpeedyCloud (article compilation)

王玉平@Ucloud (material collation)

Introduction

After a lively discussion in the group, we gained a clear understanding of “operations automation” and will focus on sharing toolsets, process‑driven engineering, and related methodology.

Outline

Build automation with modular blocks

Methodology of operations automation

Why automation tools exist

How to select automation tools

Build Automation with Modular Blocks

老王 explains his long‑time advocacy for automation, dividing automation platforms by scenario, system layer, and coupling with business programs. He also classifies platforms into three layers similar to cloudliu’s approach.

The construction starts with creating individual “blocks” and ends with assembling them.

Automation often begins with a CI platform to maximize benefits, crossing development, testing, and operations, and enforcing operational standards. It typically includes:

Standardization and规范化

Public and service‑oriented architecture

Fine‑grained, stateless services

Automation is tightly linked to standardization and business architecture; UC is already in the second stage, with some services entering the third stage.

As automation progresses, platforms become simpler, and building business‑oriented services is just a matter of assembling different blocks.

Statelessness means services do not rely on a single node; we map core business paths, identify single points, and introduce a naming service center to draw access flows, enforcing high‑availability requirements such as dual centers, degradation, and overload protection.

Team and individual capability models determine how far automation can go; the goal is to shift operations from low‑value manual work to value‑added and optimization tasks.

Key contradictions include the need for accurate CMDB data and low‑value repetitive activities; automation should address these while serving overall operational goals.

Operations Automation Methodology

Scenario‑driven: find your own use cases; open‑source solutions may not fit.

Global‑driven: set a platform‑wide goal such as intelligent operations.

Divide and conquer: design APIs early; some systems are built by other teams.

Bottom‑up: understand the foundation before building business‑oriented automation.

Plugin‑based: enable service plugins to extend the platform.

Clear boundaries: avoid overstepping, e.g., embedding scheduling into an application release system.

Why Automation Tools Exist

When a company has only a few servers, scripts suffice. As the environment grows to clusters and horizontal scaling, scripts become inefficient, prompting the emergence of dedicated automation tools.

How to Choose Automation Tools

Case 1: From Puppet to SaltStack

Initially tried Puppet for mass script deployment, but its size and rapid game iteration made it cumbersome. Explored SaltStack and Ansible; SaltStack offered quick deployment with simple installation, while Ansible required SSH key management, which was problematic at scale.

Case 2: Why Choose Ansible?

In scenarios where third‑party vendors manage servers, installing agents is undesirable. Ansible’s agent‑less SSH approach fits, allowing integration via its API into the operations platform.

Case 3: Building Our Own Tool

Complex configuration management led some teams to develop in‑house solutions using Python or Ruby, focusing on platform‑level services (DNS, LVS) while business operations handle CI and automation tasks.

How to Collaborate Effectively

InfoQ’s founder created the “InfoQ High‑Efficiency Operations” WeChat group for senior internet operations professionals, operating on an invitation‑only basis.