How 360’s OpenStack Scheduler Optimizes Multi‑Cloud Resource Allocation

Background

OpenStack is 360’s cloud solution. As the company grew, it now operates 16 regional data centers and 29 OpenStack clusters, each offering various compute resource pools (cloud‑disk, local‑disk, network‑enhanced, etc.). The environment hosts thousands of servers and tens of thousands of VMs, with continuous VM creation and deletion requests, supporting diverse business types such as pay‑as‑you‑go, elastic scaling, pre‑emptible, and deployment sets, and VM flavors including compute‑optimized, memory‑optimized, general‑purpose, local HDD/SSD, and GPU.

The platform uses a three‑layer architecture for multi‑cloud management. Users select region, zone, flavor, storage, image, network, and quantity; the request is passed to the cloud‑management layer, which forwards it to the AoChuang portal (an OpenStack access gateway) for resource control and scheduling. The bottom layer consists of the OpenStack clusters in each data center, segmented into resource pools by tags (shared, enterprise, local‑disk, network‑enhanced, etc.).

Nova‑Scheduler Resource Scheduling

When a user creates a VM, the request is first processed by nova-api , which validates the request, checks quotas, and ensures required network, image, and flavor are available. After validation, nova-api assigns a unique VM IP and records the VM state in the database, then forwards the request to nova-conductor . The conductor builds a RequestSpec object and calls the nova-scheduler select_destination API. The scheduler converts the request into a ResourceRequest , performs a pre‑filter with Placement, and makes a scheduling decision based on the latest system state, returning the chosen compute node to the conductor, which finally hands the request to the appropriate nova-compute service.

The default scheduler, FilterScheduler , uses host filtering and weighing algorithms; its available filters are configured via scheduler_available_filters . An alternative, ChanceScheduler , selects a host randomly.

AoChuang Resource Scheduling Implementation

While nova-scheduler provides precise VM placement, it lacks features such as remaining package resources, resource‑pool conversion, multi‑cluster scheduling, deployment‑set scheduling, and pre‑emptible handling for pay‑as‑you‑go packages. AoChuang implements these functions with the following process:

Check whether the image and network belong to the IDC data center and whether available IPs meet the request; the final usable region is the intersection of image, network, and network_v2‑available clusters.

Determine if the user’s package is a deployment‑set type and whether it exceeds deployment‑set limits. Deployment sets enforce strict VM distribution across physical servers for high availability.

Adjust the flavor based on user role, prioritizing network‑enhanced and single‑NUMA packages. Pay‑as‑you‑go users have four package priorities (dpdk‑single‑NUMA, dpdk‑dual‑NUMA, normal‑single‑NUMA, normal‑dual‑NUMA). Elastic‑scale users are converted to single‑NUMA packages; pre‑emptible users only support dual‑NUMA packages. Special business needs (e.g., big‑data pre‑emptible) are also handled. If the data center’s pre‑emptible instance count reaches its limit, scheduling fails; otherwise, clusters are examined further.

Iterate over OpenStack clusters in the data center, use flavor traits to select the appropriate compute resource pool, retrieve candidate hosts, and calculate each host’s ability to satisfy the flavor (CPU, memory, NUMA, ephemeral, PCI devices). This yields the number of VMs each host and cluster can create for the requested package.

For pay‑as‑you‑go requests with insufficient resources, trigger a check for pre‑emptible instance eviction; if evictable instances satisfy the demand, a displacement request is issued, giving priority to pay‑as‑you‑go users. If pre‑emptible resources are also insufficient, scheduling fails.

AoChuang’s scheduler now operates normally across all data centers, delivering NUMA‑level precision, supporting pay‑as‑you‑go, elastic scaling, and big‑data pre‑emptible workloads, creating about 3,700 VMs daily with a success rate above 99%.

Existing Problems

Resource fragmentation: uneven VM creation order leads to small‑package VMs occupying nodes, preventing placement of large‑package VMs, causing underutilized clusters.

Delayed synchronization: the cloud‑management platform does not instantly sync compute‑node resource status; it periodically pulls hypervisor data from Nova, causing scheduling inaccuracies.

Future Plans

Develop dynamic resource consolidation to migrate VMs from under‑utilized servers to those with spare capacity, maximizing node utilization.

Prioritize single‑NUMA packages for high‑performance VMs and improve dual‑NUMA distribution to balance resource usage.

Enable real‑time synchronization of OpenStack compute‑node resources to the Ultron system.