How to Achieve Seamless Test Environment Isolation with Environment Coloring in Microservices

Background

In a microservice architecture, rapid business growth leads to an ever‑increasing number of services, massive dependencies, and long call chains. Parallel development creates high costs for environment isolation: the existing gateway routing cannot perform dynamic routing, branch deployments are tightly coupled with clusters, physical isolation requires costly instance scaling, and managing multiple Nginx/API‑gateway configurations and host switches becomes cumbersome.

Objectives

Establish a stable development environment by defining a unified domain‑name convention, eliminating the need for per‑environment Nginx/API‑gateway configurations and host switches, and enabling zero‑configuration test‑environment provisioning.

Create a reliable baseline test environment to improve stability and accessibility, and define access standards for development, integration, feature, and regression environments.

Build a reasonable development workflow: support dynamic routing via service discovery, remove the need for physical cluster isolation, and provide one‑click environment‑isolation deployment through the R&D collaboration platform.

Reduce troubleshooting cost by isolating messages and ensuring that colored requests are never consumed by the wrong environment.

Solution Overview

A lightweight environment‑coloring solution was designed. A color tag is attached to service instances; the tag travels with each request (HTTP(s), client WZP, or message) and is used by the gateway and routing layer to match the appropriate instance. If no matching colored instance exists, the request falls back to an uncolored (baseline) instance.

Core Mechanism

When a request is made, the client includes the color tag in a custom header (e.g., X‑YX‑COLOR). The gateway extracts the tag, forwards it to downstream services, and the routing component first searches for instances with the same tag across all clusters. If none are found, it selects an uncolored instance in the current cluster. This ensures strict isolation for colored traffic while allowing uncolored traffic to use the baseline environment.

Scope of Coloring

Phase 1: Apply coloring to HTTP(s) and client‑side WZP requests.

MPS (Yanxuan Message Publishing‑Subscription system) supports coloring to achieve message consumption isolation.

Coloring is limited to test environments; regression environments remain uncolored.

Placement Principle

Only services that participate in a specific feature test need to be colored. All other services stay uncolored and will automatically receive traffic from colored requests as fallback, achieving isolation without replicating the entire service mesh.

Implementation Details

Instance Binding of Color Tags

The service‑governance and deployment platforms allow setting color tags at the cluster or instance level. Multiple tags can be bound to a single instance, and the relationship is persisted.

Domain‑Name Planning for Coloring

Old test environments required separate domain names, Nginx configs, and API‑gateway rules for each environment. The new plan introduces a wildcard domain such as .feature.you.163.com, where the asterisk part holds the color tag value. Only a single set of Nginx/API‑gateway configurations is needed.

Gateway Parsing of Colored Domains

The test Nginx gateway listens for the colored domain, extracts the color string from the wildcard position, sets a new HTTP header X‑YX‑COLOR: <color>, and forwards the request to the Yanxuan API gateway, preserving the header if it already exists.

Propagation of Color Tags Between Services

CaesarAgent, the distributed tracing agent, reuses its trace‑ID propagation capability to add the X‑YX‑COLOR header to outbound HTTP calls. This enables automatic tag transmission across service boundaries.

Routing Logic

For uncolored requests (no X‑YX‑COLOR header), the router searches the current cluster for all forwardable nodes, including colored ones, and distributes traffic proportionally based on weight. Colored instances should not share the default test cluster to avoid cross‑contamination.

For colored requests, the router first looks across all clusters for instances with the matching color tag. If none are found, it falls back to an uncolored instance in the current cluster; if no uncolored instance exists, the request fails with a 502 error.

The rule can be summarized as: match on Consul tag and color value, with color taking precedence; colored requests may cross clusters, uncolored requests cannot.

Health‑Status Handling

Even if a colored instance is unhealthy, it is still considered a match for routing. If only unhealthy colored instances exist, the router returns 502 instead of falling back to an uncolored node, preserving isolation guarantees.

Message Isolation Mechanism

MPS, the internal message‑publishing system, already uses HTTP/Consul calls, so the same X‑YX‑COLOR header can be attached to publish requests. Subscriptions receive messages via push calls, automatically inheriting the routing logic without code changes.

Mobile WZP Coloring

Mobile clients use the WZP protocol. The new scheme binds a device ID to a color tag, lets the API gateway extract the device ID, map it to a color, and inject X‑YX‑COLOR into the request header, achieving end‑to‑end coloring for all mobile protocols.

TianShu One‑Click Isolation

The TianShu platform (Yanxuan’s one‑stop R&D collaboration tool) provides a one‑click interface to bind color tags to instances during deployment, automatically remove tags after branch merge, and keep the environment clean for future feature testing.

Asynchronous Thread Coloring

In asynchronous scenarios (e.g., third‑party payment callbacks or async HTTP calls), the color tag may be lost. The solution recommends adding the yanxuan‑trace‑client dependency and explicitly retrieving the tag via TracerClient.getTagItem("x‑yx‑color") before making async calls.

// Check if current environment is a test environment and the color tag is not empty
if (systemEnvironmentConfig.isTestEnvironment() && StringUtils.isNotBlank(TracerClient.getTagItem("x-yx-color"))) {
    // Retrieve the color value
    String color = TracerClient.getTagItem("x-yx-color");
    // TODO: use the color value
} else {
    // TODO: handle non‑test or non‑colored request
}

Future Plans

Unify client‑side coloring capabilities across all app protocols for multi‑environment switching.

Decouple client testing from internal network and Wi‑Fi dependencies by embedding multiple host configurations for online, pre‑release, regression, and test gateways.

Enable coloring for scheduled tasks.

Conclusion

R&D efficiency follows the “barrel principle”: a shortage in any part reduces overall output. Environment isolation is a critical component of Yanxuan’s efficiency system, and the article thanks the foundational technology, client, frontend, and backend teams for their support in building a robust testing environment.