How We Built a Multi‑Task Scheduler with Mesos on Legacy Servers

Background

Our internal cloud platform provides many physical and virtual machines, but we observed low CPU and memory utilization. By fully exploiting idle resources while keeping services running, we could save considerable hardware costs, prompting us to explore Mesos for a multi‑task scheduling system.

Why Not Kubernetes

Most people would choose Kubernetes + Docker, but our production machines run CentOS 6.2 with Linux kernel 2.6, which cannot run Docker (requires kernel 3.10+). Upgrading the kernel or rebooting machines is not feasible, so we turned to Mesos.

Mesos Overview

Mesos is described as an operating system for the entire data center, managing CPU, memory, disk, and network resources, allocating them per task, handling scheduling, and supporting fault‑tolerance.

Features

Mesos’ key feature is two‑level resource scheduling, illustrated below:

Each Agent reports its resources to the Master.

The Master sends resource offers to Frameworks.

Frameworks match pending tasks with offers and reply to the Master.

The Master forwards the matched tasks and offers to the appropriate Agents.

Agents run tasks via Executors with enforced resource limits.

Mesos can run frameworks such as Spark, Storm, Hadoop, and Marathon.

System Architecture

The overall architecture is shown below:

Key components:

Mesos Master + Mesos Agents.

Marathon as the second‑level scheduler.

A custom Monitor deployed alongside each Agent to adjust resources in real time.

Workflow:

Monitor collects available resources on each machine.

Monitor updates the reserved resources for each Agent.

Agents report their reserved resources to the Master.

Master forwards resource offers to Marathon.

Marathon allocates tasks based on offers and returns results to the Master.

Master dispatches tasks to the selected Agents.

Challenges

Deploy Mesos agents on business machines without intrusive changes.

Real‑time monitoring and dynamic adjustment of Agent resources.

Fast task deployment and strict resource isolation.

Cluster‑wide operational monitoring.

Implementation Details

We use Mesos 1.4.1 written in C++11. To avoid polluting the host environment, all required libraries are placed in a dedicated libs directory, and the Mesos binary’s rpath is patched with patchelf to point to this directory.

Compilation steps follow the official guide ( http://mesos.apache.org/documentation/latest/building/ ), with adjustments such as using GCC 5.4 on CentOS 6.2 and skipping JavaDoc generation to avoid missing protobuf jars.

Our custom Monitor runs on each Agent, periodically measuring free resources and adjusting the dynamic resource reservation via Mesos’s HTTP API, eliminating the need to restart Agents.

Task isolation relies on Mesos Containerizer (using Linux cgroups and namespaces) and, where kernel limitations exist, a rootfs + chroot sandbox. We package tasks with appropriate rootfs images and store them in S3 for Marathon to deploy.

CPU isolation can be configured as either shared (no hard limit) or CFS‑based hard limits. Disk isolation uses periodic du checks.

For monitoring, we deploy mesos‑exporter alongside the Master/Agents, push metrics to Prometheus, and visualize them with Grafana. Mesos also provides extensive HTTP APIs for custom monitoring.

Remaining Issues

Task packaging is still manual, requiring custom rootfs creation for each language/runtime, which is time‑consuming and error‑prone. Additionally, we only use Marathon, which suits long‑running services; we lack support for scheduled (cron‑like) tasks.

Conclusion

We have introduced our Mesos‑based multi‑task scheduling system, covering design, deployment, monitoring, and challenges, and invite interested peers to discuss and improve the solution.