Design and Evolution of Shopee Image Service: Architecture, Challenges, and Optimizations

1. Background

Images are a critical multimedia resource whose quality directly impacts user experience. In Shopee, virtually every scenario—product listings, reviews, user avatars, icons—relies on images. To consolidate image traffic and standardize processing, Shopee built a unified, one‑stop Image Service platform that provides high‑quality upload, processing, and delivery capabilities.

2. Core Challenges

Challenge 1: Legacy business systems are fragmented, making integration costly and complex.

Challenge 2: Overseas network conditions are heterogeneous, threatening upload/download stability and availability.

Challenge 3: While keeping access logic lightweight, the service must still schedule image quality based on user network, device type, and other dimensions.

3. Shopee Image Service Technical Solution

3.1 Overall Design

The platform integrates upload, processing, and publishing into a single service, offering broad applicability and flexibility. It covers the full chain from content creation, through the server side, to end‑user consumption.

3.1.1 Functional Modules

Acquisition & Upload: Processing SDK (compression, cropping, transcoding) and Upload SDK with resumable and pull‑upload mechanisms, plus CDN‑level transmission acceleration.

Media Management & Production: Metadata management, fingerprint library, hotspot data, image processing (basic scaling, cropping, format conversion, blind watermarking, quality enhancement, scoring), and storage migration operators.

Distribution & Access: Client SDK for URL scheduling, network loading, decoding, and quality reporting; distribution control for anti‑hotlinking, referer checks, CDN pre‑warming, and per‑device CDN/format selection.

Each business receives a unique Business ID, ensuring isolation of upload, processing, and downstream configurations.

3.1.2 Image ID Design

ImageID must support domain convergence and optional custom paths. Two mechanisms are provided:

Universal ID: Server‑generated key containing Business ID; URLs are built by concatenating a common domain with the ID.

Custom Rule: Users follow standard S3 upload semantics, specifying bucket and object; the server returns a unique ID.

The combination satisfies both generality and extensibility.

3.2 Image Upload Optimization

3.2.1 Link‑Level Optimizations

Cache upload credentials and IDs on the client to reduce repeated requests.

Leverage storage‑side event notifications to replace client‑to‑server signaling with server‑to‑server calls, reducing latency and failure rates.

Introduce an image‑access proxy that defers sub‑image generation to access time, reducing upload‑time coupling and storage waste.

The revised link flow is illustrated in the original diagram (Fig. 8).

3.2.2 Transport‑Level Optimizations

Mitigate single‑source storage bottlenecks by using commercial CDN nodes, configuring lifecycle policies, and performing asynchronous migration.

Employ full‑site acceleration CDN to shorten request paths from Southeast Asian data centers.

Adopt HTTP/3 + QUIC for upload transport, improving performance in weak network environments.

These measures form a closed‑loop of data reporting, policy generation, and optimal dispatch (Fig. 10).

3.2.3 Additional Enhancements

Fingerprint library to deduplicate uploads and avoid unnecessary transfers.

IP‑direct return to bypass DNS resolution failures.

3.3 Image Access Design

Original access logic hard‑coded domains and concatenated ImageIDs, leading to limited CDN scheduling, format selection, and high integration cost.

3.3.1 Access Scheduling

A cloud‑configuration component (云配) distributes rules based on app version, device model, network, and geographic location, enabling dynamic selection of image format (e.g., WEBP, HEIF) and optimal CDN domain.

3.3.2 Downstream Service

Image Proxy provides real‑time processing via URL suffixes. The architecture consists of three layers:

Filter: Request validation, key parsing, and rate limiting.

Upstream: Routing and load balancing to storage origins.

Handler: Core logic including cache management, task flow, image processors (Local‑ImgLibs, FPGA Cluster, AI Cluster), IQA assessment, and reporting.

Heavy‑weight tasks (large GIF scaling, high‑resolution WEBP conversion, AI enhancement) initially return the original image with short CDN cache, then trigger asynchronous processing.

3.3.3 Adaptive Image Size

Based on device screen size, the service performs adaptive resizing to reduce bandwidth, download time, and memory consumption.

4. Technical Achievements

Upload optimization: Daily uploads reach billions; success rate improved from 96.5 % to 99.67 %; average upload latency reduced by 50 %.

Storage and bandwidth: Real‑time processing saved >25 % storage; adopting high‑compression formats (WEBP, etc.) cut bandwidth by 56 %.

App performance: Adaptive compression reduced image size by 75 %, download time by 20 %, failure rate by 0.7 %, and decode time by 30 %.

5. Summary and Outlook

After more than a year of collaboration, Shopee’s Image Service now handles the majority of image traffic, providing unified governance and optimization across upload, processing, and distribution. Future work includes developing proprietary codecs for edge devices, expanding OpenAPI capabilities, and adding more intelligent production and content‑security features.