Designing Queueing and Rate Limiting for Scalable AIGC Services

1. Product Behaviors

1.1 Queue System Product Behaviors

The queue system handles requests that cannot be processed immediately, informing users that their command has been received and is being processed, typically through waiting indicators, status updates, and asynchronous notifications.

Spinning wheel and progress bar: When a user submits a request (e.g., generate an image), a loading animation or an approximate progress bar appears, indicating "processing". Behavior: The UI shows a loading animation or an imprecise progress bar after the request enters the backend queue awaiting GPU resources. Logic: The request is queued; the animation tells the user to wait while the backend processes it.

Explicit "Queued" or "Processing" status: Behavior: For longer tasks (e.g., multi‑minute video generation), the UI may show statuses like "Queued (position 5)", "Processing (3 minutes left)", "Completed" in a "My Tasks" list. Logic: This reflects the queue and processing unit status; users can leave the page and return later.

Asynchronous notification: Behavior: After submitting a task, the system says "Task submitted, you will be notified when done"; later the user receives an in‑app push, email, SMS, etc., with the result. Logic: Typical async processing + queue: the request is queued, the UI responds, and the result is pushed after completion.

Estimated waiting time: Behavior: Some products display an estimated wait time based on current queue length and processing speed, e.g., "Estimated wait: ~5 minutes". Logic: The system monitors queue status and predicts wait time using historical data or current load.

Temporarily unable to submit new tasks: Behavior: In extreme peaks, the product may block new tasks of the same type and show "System busy, please try later". Logic: This protection prevents unlimited queue growth and system crashes.

Point 5’s temporary block is actually a rate‑limiting strategy.

1.2 Rate‑Limiting Product Behaviors

Rate limiting protects the system from being overwhelmed, so its manifestations relate to service denial, error prompts, and quota display.

"Too fast, try later": Behavior: Rapid clicks or a high‑frequency script trigger errors like "Operation too frequent, try later", "API call limit reached", or HTTP 429. Logic: The request exceeds a rate‑limit rule (e.g., 10 calls per minute); the server rejects the excess.

Human verification required: Behavior: Sensitive actions (login, posting) trigger image captchas, slide verification, or reCAPTCHA. Logic: Anti‑scraping/anti‑spam measure; the system suspects a bot and adds a hurdle.

Feature temporarily unavailable or degraded: Behavior: Free AIGC tools may limit daily image generations; after the limit, the generate button is disabled or a prompt to upgrade appears. During peaks, free users may receive lower‑resolution images or "degraded intelligence" results. Logic: Quota‑based rate limiting differentiates users and guides them to paid plans.

Explicit quota/usage display: Behavior: Account settings or API console show usage, e.g., "API calls this month: 850/1000", "Images generated today: 3/5", "Parallel queues: 5". Logic: Transparent quota display lets users plan usage and avoid sudden rejections.

1.3 Product Behavior Summary

Queue system mainly manages waiting expectations and provides status feedback to smooth long‑running tasks and improve user experience.

Rate limiting mainly uses explicit rejections or restrictions to protect the system, ensure fairness, control costs, and sometimes serve as a commercial differentiator.

Rate limiting is the entrance guard deciding "if you can enter" and "how fast you can enter".

The queue system is the waiting‑area manager handling "who is waiting and how they are ordered".

2. Design Considerations

Beyond knowing users may see spinners or "try later" messages, designers must consider many factors when choosing queue and rate‑limit strategies, similar to planning a large event: balance user experience, venue capacity, resource consumption, and VIP treatment.

2.1 Goal 1: What effect do we want?

The primary reasons for adding queueing and rate limiting are:

Survival: Prevent system crash under sudden high concurrency (e.g., DeepSeek outage).

Cost control: GPU inference is expensive; limit total calls and smooth resource usage.

User experience: Define acceptable wait times, decide between fast results or guaranteed completion.

Fairness & differentiation: Offer higher limits or priority to paid or high‑tier users.

Abuse prevention: Stop malicious scraping or low‑value bulk calls via IP/user‑based limits and captchas.

Clear goals guide subsequent design.

2.2 Tailor to system and business

There is no one‑size‑fits‑all solution; design must match characteristics:

Task characteristics: Varying duration (seconds to days), resource consumption (CPU/GPU/memory), parallelism.

Traffic patterns: Steady, tidal, or bursty (events, hot topics).

Technical stack & infrastructure: Cloud (AWS, Azure, GCP) vs. self‑hosted, monolith vs. microservices.

Business model: Freemium vs. pay‑per‑use influences quota and limit policies.

2.3 Queue strategies

FIFO: Simple, fair, suitable for most cases.

Priority queue: Allows VIP or urgent tasks to jump ahead; more complex.

Delay queue: Defers tasks to a later time, useful for retries or scheduled jobs.

Decide whether to use a single large queue or multiple queues per task type or user tier.

Consider persistence (e.g., Kafka, RabbitMQ durable mode) and dead‑letter queues for failed tasks.

2.4 Rate‑limiting strategies

Algorithms: Token bucket (burst allowed), leaky bucket (smooth rate), fixed/sliding window counters.

Dimensions: Per user/API key, per IP, per endpoint, per model, global.

Placement: Gateway layer (API Gateway, Nginx, Kong), service layer, middleware/library.

Post‑limit actions: Immediate rejection (429), short internal buffer queue, degraded response (fallback model).

2.5 User experience

Transparency: Show clear status, progress bars, estimated wait times.

When limited: Friendly error messages explaining reason and retry time, plus documentation.

Quota display: Show current usage vs. total quota.

Expectation management: Provide estimates to set user expectations.

Graceful error handling: Offer guidance instead of raw error codes.

2.6 Monitoring & iteration

Metrics: Queue length, average wait, backlog, consumer throughput, dead‑letter count; rate‑limit request count, blocked count, rule distribution; system CPU/GPU, memory, network, error rate.

Alerts: Trigger when thresholds (e.g., long queues, high reject rate) are exceeded.

Tuning: Adjust limits, queue priorities, consumer numbers based on data; use A/B testing for strategy validation.

3. Technical Implementation

After defining "what" and "why", we need to choose tools and methods to implement queueing and rate limiting.

3.1 Queue technology selection

We usually adopt mature message‑queue middleware rather than building from scratch.

RabbitMQ: Mature, supports many protocols, flexible routing, good for complex routing and reliability.

Kafka: Extremely high throughput, persistent log‑style queue, suited for massive request volumes and replay.

Redis: In‑memory, fast; use List (LPUSH/RPOP) for simple queues or Streams for richer features; good if Redis is already in the stack.

Cloud‑provider MQ services: Managed services (AWS SQS, GCP Pub/Sub, etc.) require minimal ops, integrate well with other cloud resources.

Choosing guidelines:

Complex routing → RabbitMQ.

Ultra‑high throughput → Kafka.

Simple, already using Redis → Redis Lists/Streams.

Fully managed cloud → Cloud MQ.

Implementation notes:

Producer: API service packages user request (prompt, user ID, priority) into a message and pushes to the queue.

Consumer: Worker processes pull messages, execute the AIGC task, acknowledge (Ack) on success, or move to dead‑letter on failure.

Concurrency control: Scale consumer instances but keep GPU usage within limits.

Message design: Keep payload small; store large data (e.g., images) elsewhere and reference via URL.

3.2 Rate‑limiting technology selection

Rate limiting can be placed at various layers.

Gateway layer: Nginx (limit_req), Kong, Apigee, AWS API Gateway, etc.; provides unified, high‑performance protection.

Application / code layer: Use language‑specific libraries (Java Guava RateLimiter, Go x/time/rate, Python ratelimiter, Node express-rate-limit) or framework middleware for fine‑grained control.

Pros/cons:

Gateway: Centralized, low overhead, but less flexible for complex business logic.

Code layer: Highly flexible, can incorporate user tier, request parameters, but requires implementation in each service.

State storage for distributed rate limiting:

Redis: Fast, atomic operations (INCR, EXPIRE) and Lua scripts ensure consistency across instances.

In‑memory: Suitable only for single‑instance services.

Database: Generally too slow for high‑frequency checks.

Conceptual implementations:

Token bucket (Redis + Lua)

Store token count and last refill timestamp per user/key.

On request, compute tokens to add based on elapsed time, cap at bucket size, update count and timestamp.

If token count > 0, decrement and allow request; else reject.

Sliding window log (Redis Sorted Set)

Each request adds a member with current timestamp as score.

Remove entries older than (now – window size).

Count remaining entries; if below limit, allow, otherwise reject.

3.3 Integrating into AIGC workflow

User sends request (e.g., click "Generate Image").

Entry rate limiter checks user/IP quota; rejects with 429 if exceeded.

If allowed, backend validates and creates a task message.

Message is sent to the chosen MQ; UI can show "Task submitted, queued/processing".

Message waits in queue according to FIFO or priority.

Worker (consumer) pulls message, optionally applies internal rate limits for downstream resources.

Worker runs the AIGC model, generates result.

Result is stored (e.g., object storage) and a completion acknowledgment is sent to the queue.

User is notified via WebSocket, callback URL, or polling.

Rate limiting mainly protects the entry point and sometimes downstream resource calls, while the queue provides buffering, async decoupling, and reliable processing.

4. Summary

For AIGC architectures, queue systems and rate‑limiting strategies are not optional; they are core components ensuring stability, availability, fairness, and cost‑effectiveness. During design you must:

Identify bottlenecks (typically model inference).

Define policies based on user experience, cost, and fairness goals.

Select appropriate tools (queue middleware, rate‑limit components) matching your stack and performance needs.

Continuously monitor queue lengths, wait times, rate‑limit triggers, and system load, and iteratively tune parameters.

These practices keep AIGC services robust under massive user demand.