Rules of Machine Learning: 43 Practical Guidelines for Building Robust ML Systems

The article, originally written by Martin Zinkevich, presents 43 practical rules for developing and optimizing machine‑learning‑based product systems, aimed at engineers who have basic ML knowledge.

Terminology

Key terms are defined without translation to preserve their original meaning: Instance (an unlabeled sample), Label (the answer), Feature (a property of an instance), Feature Column (a set of related features), Example (instance + label), Model , Metric , Objective , and Pipeline .

Overview

Great products are built by engineers first, not by ML experts; most gains come from good features rather than sophisticated algorithms. The core approach is to ensure a solid end‑to‑end pipeline, start with a reasonable objective, add common‑sense features simply, and keep the pipeline robust.

Before Machine Learning

Don’t be afraid to launch a product without ML.

Design and implement metrics before introducing ML.

Replace overly complex heuristics with ML models when appropriate.

ML Phase I: Your First Pipeline

Keep the first model simple and get the infrastructure right.

Test the infrastructure independently from the ML code.

Be careful about dropped data when copying pipelines.

Turn heuristics into features or handle them externally.

Monitoring

Visual monitoring and alerting are essential for efficient operation.

Training‑Serving Skew

Differences between offline training and online serving arise from data processing, feature drift, and feedback loops. To minimize skew, log exactly the features used at serving time and reuse them for training.

ML Phase II: Feature Engineering

After a stable pipeline, focus on expanding and iterating over features. Guidelines include using directly observed features first, exploring cross‑context features, preferring specific over generic features, combining features in human‑understandable ways, and cleaning up unused features.

Human Analysis of the System

Engineers should not assume they are typical end users; instead, gather real‑user feedback. Measure model deltas, prioritize utilitarian performance, and create new features from error patterns.

Training‑Serving Consistency

Reuse code between training and serving pipelines, test models on data later than the training window, and be aware of feedback loops caused by positional features.

ML Phase III: Slowed Growth, Optimization Refinement, and Complex Models

When growth plateaus, avoid adding new features unless objectives are clear, keep ensembles simple, seek qualitatively new information sources, and recognize that diversity, personalization, and relevance are not always correlated with popularity.

Summary

The 43 rules provide a comprehensive, experience‑driven checklist that helps teams build, monitor, and evolve ML systems while staying grounded in engineering best practices.