Implementing Test Right-Shift in Game Development: Stable Release, Monitoring, and Risk Control

With the gradual adoption of agile practices, the role of QA has evolved from simple functional testing to a broader scope that includes both test left‑shift and test right‑shift. This article examines how the test right‑shift concept is applied within a game development team, focusing on stable release, monitoring, and risk control.

Stable Release : The team uses a gray‑release approach where new content or engine updates are first deployed to a small subset of servers or clients. Feature toggles are added at logical entry points, allowing selective activation. On the client side, multiple executable versions coexist, and a probability file determines which version runs. Server‑side updates follow a staged rollout, starting with a limited set of servers before full deployment.

Log Monitoring : Player behavior logs are stored in MongoDB and displayed via a custom data‑center UI, enabling detailed queries without SQL knowledge. For macro‑level analysis, a third‑party platform (Fuxi) provides SQL‑based access to logs. An automated log‑monitoring system runs Python scripts on scheduled tasks, pushes alerts through an internal IM bot (POPO), and supports custom monitoring scripts such as attribute‑snapshot checks and whitelist‑based detailed combat logs.

Opinion Monitoring : The team combines passive bug reports from operations with proactive scanning of player discussions across forums, social media, and in‑game chat. A keyword‑triggered workflow routes operation‑submitted bugs to a centralized bug‑tracking platform, while automated tools collect and score external feedback, feeding high‑risk items into the QA pipeline.

Performance Monitoring : Two internal tools—Postman (visual performance dashboards) and Monitor (CPU/memory metrics)—provide real‑time visibility into server health. A case study describes how a latency issue was traced to a disk I/O problem after comparing performance metrics across servers. Additional monitoring for database managers (dbmgr) captures long‑running SQL queries and feeds alerts to the POPO bot, with plans to visualize data via Kibana.

Risk Control : Predictive control mechanisms trigger alerts or block actions when game metrics exceed predefined thresholds. Examples include first‑clear alerts for new dungeons, abnormal attribute or damage values, and prolonged invincibility states. The team also designs item‑distribution limits that issue warnings or block excess rewards, with automated rollback commands for rapid remediation.

In conclusion, the goal of test right‑shift is to enable faster feedback loops between the technical and business sides, fostering a culture where quality assurance is integrated throughout design and development. By sharing tools, processes, and lessons learned, the team aims to continuously improve the robustness of the game’s live operations.