User Experience Analysis Framework and Metric Design for E‑commerce Platforms

This article is the first of a ten‑part series that shares practical experience and research on user‑experience data science at Taobao, covering product detail pages, logistics, performance, messaging, customer service, and journey analysis.

The authors present an overall analysis framework, likening data scientists to doctors: discover problems (experience), diagnose (clinic), devise treatment strategies (therapy), and verify effects (follow‑up). The process is applied to improve browsing decisions, logistics satisfaction, and performance metrics.

A core tool is the User Experience Map (UEM), which structures the journey into behavior stages, user goals, actions, needs, pain points, and opportunities. The map helps identify key moments (MOT) and guides systematic, data‑driven optimization.

Metric design follows a two‑layer model: X‑metrics (experience‑oriented, e.g., NPS, satisfaction, VOC) and O‑metrics (operational, e.g., DAU, conversion, retention). The design steps include: (1) mapping the user journey, (2) defining experience metrics at the surface, line, and point levels, and (3) creating corresponding operational metrics. Principles emphasize focusing on MOT, aligning experience and operational metrics, and iterating designs based on business insights.

The article also covers AB testing and causal inference. It explains hypothesis formulation, sample‑size calculation, uniform traffic allocation, monitoring, and result analysis. A Python example for power analysis is provided:

from statsmodels.stats.power import zt_ind_solve_power
from statsmodels.stats.proportion import proportion_effectsize as es
zt_ind_solve_power(effect_size=es(prop1=0.30, prop2=0.305), alpha=0.05, power=0.8, alternative="two-sided")

Additional methods such as propensity‑score matching (PSM) for non‑randomized experiments are discussed, along with practical guidance on metric thresholds, strategy rollout, and validation.