Building Machine Learning Systems in Small Teams: Practices, Pitfalls, and Lessons from Dangdang

Machine learning systems are powerful but difficult to build reliably, especially for small teams. The speaker explains the motivation for a process‑oriented sharing of practical experiences, focusing on the unique constraints and opportunities of a compact ML team at Dangdang.

Small‑Team Perspective

Small teams face high uncertainty in ML projects, scarce talent, and the need for each member to handle multiple responsibilities. While this creates challenges such as high individual workload, cross‑functional task ownership, careful prioritization, and increased single‑point risk, it also brings advantages like easier cohesion, rapid collaboration, faster iteration, and accelerated personal growth.

Dangdang Recommendation ML Practice

The team is responsible for end‑to‑end development, tuning, maintenance, and improvement of ML pipelines for recommendation and advertising ranking. Early stages involve exploratory work using R and Python to validate problem suitability. As data volume grows, they transition to Hadoop and Spark for large‑scale processing.

The development workflow mirrors building a house: lay the foundation, construct a shell, then continuously renovate through validation, full‑process construction, and iterative optimization (offline tests, online A/B). Tools include common industry solutions and a proprietary feature‑engineering suite called dmilch (Dangdang Machine Learning toolChain).

Key Process Insights

Key practices include maintaining seriality of changes to isolate effects, holding weekly meetings for rapid discussion and decision‑making, encouraging team participation, and cautiously adopting new technologies only after mastering existing ones.

Pitfalls Encountered

Focusing solely on model metrics without considering system integration leads to “model‑only” solutions that fail in production.

Neglecting visualization and diagnostic tools makes debugging difficult; the team built a web UI to inspect samples, features, and rankings.

Over‑reliance on algorithms; sometimes simple manual filtering outperforms complex models.

Relying on external teams for critical data pipelines can cause data quality issues; taking ownership of data collection improves reliability.

Lack of full‑stack expertise (e.g., missing front‑end talent) hampers end‑to‑end problem solving.

Creating a monolithic "giant" system leads to high coupling and maintenance pain; modularization and “big system, small implementation” mitigates this.

Future Challenges

The team reflects on technical debt in ML systems (as described in a Google Research paper) and the broader challenge that ML breaks traditional software engineering practices, requiring new roles such as ML system architects.

Overall, the presentation offers practical guidance for small ML teams aiming to build robust, scalable recommendation systems while avoiding common traps.